{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 个性化推荐\n",
    "本项目使用文本卷积神经网络，并使用[`MovieLens`](https://grouplens.org/datasets/movielens/)数据集完成电影推荐的任务。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "推荐系统在日常的网络应用中无处不在，比如网上购物、网上买书、新闻app、社交网络、音乐网站、电影网站等等等等，有人的地方就有推荐。根据个人的喜好，相同喜好人群的习惯等信息进行个性化的内容推荐。比如打开新闻类的app，因为有了个性化的内容，每个人看到的新闻首页都是不一样的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这当然是很有用的，在信息爆炸的今天，获取信息的途径和方式多种多样，人们花费时间最多的不再是去哪获取信息，而是要在众多的信息中寻找自己感兴趣的，这就是信息超载问题。为了解决这个问题，推荐系统应运而生。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "协同过滤是推荐系统应用较广泛的技术，该方法搜集用户的历史记录、个人喜好等信息，计算与其他用户的相似度，利用相似用户的评价来预测目标用户对特定项目的喜好程度。优点是会给用户推荐未浏览过的项目，缺点呢，对于新用户来说，没有任何与商品的交互记录和个人喜好等信息，存在冷启动问题，导致模型无法找到相似的用户或商品。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了解决冷启动的问题，通常的做法是对于刚注册的用户，要求用户先选择自己感兴趣的话题、群组、商品、性格、喜欢的音乐类型等信息，比如豆瓣FM：\n",
    "<img src=\"assets/IMG_6242_300.PNG\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 下载数据集\n",
    "运行下面代码把[`数据集`](http://files.grouplens.org/datasets/movielens/ml-1m.zip)下载下来"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "from collections import Counter\n",
    "import tensorflow as tf\n",
    "\n",
    "import os\n",
    "import pickle\n",
    "import re\n",
    "from tensorflow.python.ops import math_ops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve\n",
    "from os.path import isfile, isdir\n",
    "from tqdm import tqdm\n",
    "import zipfile\n",
    "import hashlib\n",
    "\n",
    "def _unzip(save_path, _, database_name, data_path):\n",
    "    \"\"\"\n",
    "    Unzip wrapper with the same interface as _ungzip\n",
    "    :param save_path: The path of the gzip files\n",
    "    :param database_name: Name of database\n",
    "    :param data_path: Path to extract to\n",
    "    :param _: HACK - Used to have to same interface as _ungzip\n",
    "    \"\"\"\n",
    "    print('Extracting {}...'.format(database_name))\n",
    "    with zipfile.ZipFile(save_path) as zf:\n",
    "        zf.extractall(data_path)\n",
    "\n",
    "def download_extract(database_name, data_path):\n",
    "    \"\"\"\n",
    "    Download and extract database \n",
    "    :param database_name: Database name\n",
    "    \"\"\"\n",
    "    DATASET_ML1M = 'ml-1m'\n",
    "\n",
    "    if database_name == DATASET_ML1M:\n",
    "        url = 'http://files.grouplens.org/datasets/movielens/ml-1m.zip'\n",
    "        hash_code = 'c4d9eecfca2ab87c1945afe126590906'\n",
    "        extract_path = os.path.join(data_path, 'ml-1m')\n",
    "        save_path = os.path.join(data_path, 'ml-1m.zip')\n",
    "        extract_fn = _unzip\n",
    "\n",
    "    if os.path.exists(extract_path):\n",
    "        print('Found {} Data'.format(database_name))\n",
    "        return\n",
    "\n",
    "    if not os.path.exists(data_path):\n",
    "        os.makedirs(data_path)\n",
    "\n",
    "    if not os.path.exists(save_path):\n",
    "        with DLProgress(unit='B', unit_scale=True, miniters=1, desc='Downloading {}'.format(database_name)) as pbar:\n",
    "            urlretrieve(\n",
    "                url,\n",
    "                save_path,\n",
    "                pbar.hook)\n",
    "\n",
    "    assert hashlib.md5(open(save_path, 'rb').read()).hexdigest() == hash_code, \\\n",
    "        '{} file is corrupted.  Remove the file and try again.'.format(save_path)\n",
    "\n",
    "    os.makedirs(extract_path)\n",
    "    try:\n",
    "        extract_fn(save_path, extract_path, database_name, data_path)\n",
    "    except Exception as err:\n",
    "        shutil.rmtree(extract_path)  # Remove extraction folder if there is an error\n",
    "        raise err\n",
    "\n",
    "    print('Done.')\n",
    "    # Remove compressed data\n",
    "#     os.remove(save_path)\n",
    "\n",
    "class DLProgress(tqdm):\n",
    "    \"\"\"\n",
    "    Handle Progress Bar while Downloading\n",
    "    \"\"\"\n",
    "    last_block = 0\n",
    "\n",
    "    def hook(self, block_num=1, block_size=1, total_size=None):\n",
    "        \"\"\"\n",
    "        A hook function that will be called once on establishment of the network connection and\n",
    "        once after each block read thereafter.\n",
    "        :param block_num: A count of blocks transferred so far\n",
    "        :param block_size: Block size in bytes\n",
    "        :param total_size: The total size of the file. This may be -1 on older FTP servers which do not return\n",
    "                            a file size in response to a retrieval request.\n",
    "        \"\"\"\n",
    "        self.total = total_size\n",
    "        self.update((block_num - self.last_block) * block_size)\n",
    "        self.last_block = block_num"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found ml-1m Data\n"
     ]
    }
   ],
   "source": [
    "data_dir = './'\n",
    "download_extract('ml-1m', data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 先来看看数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本项目使用的是MovieLens 1M 数据集，包含6000个用户在近4000部电影上的1亿条评论。\n",
    "\n",
    "数据集分为三个文件：用户数据users.dat，电影数据movies.dat和评分数据ratings.dat。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用户数据\n",
    "分别有用户ID、性别、年龄、职业ID和邮编等字段。\n",
    "\n",
    "数据中的格式：UserID::Gender::Age::Occupation::Zip-code\n",
    "\n",
    "- Gender is denoted by a \"M\" for male and \"F\" for female\n",
    "- Age is chosen from the following ranges:\n",
    "\n",
    "\t*  1:  \"Under 18\"\n",
    "\t* 18:  \"18-24\"\n",
    "\t* 25:  \"25-34\"\n",
    "\t* 35:  \"35-44\"\n",
    "\t* 45:  \"45-49\"\n",
    "\t* 50:  \"50-55\"\n",
    "\t* 56:  \"56+\"\n",
    "\n",
    "- Occupation is chosen from the following choices:\n",
    "\n",
    "\t*  0:  \"other\" or not specified\n",
    "\t*  1:  \"academic/educator\"\n",
    "\t*  2:  \"artist\"\n",
    "\t*  3:  \"clerical/admin\"\n",
    "\t*  4:  \"college/grad student\"\n",
    "\t*  5:  \"customer service\"\n",
    "\t*  6:  \"doctor/health care\"\n",
    "\t*  7:  \"executive/managerial\"\n",
    "\t*  8:  \"farmer\"\n",
    "\t*  9:  \"homemaker\"\n",
    "\t* 10:  \"K-12 student\"\n",
    "\t* 11:  \"lawyer\"\n",
    "\t* 12:  \"programmer\"\n",
    "\t* 13:  \"retired\"\n",
    "\t* 14:  \"sales/marketing\"\n",
    "\t* 15:  \"scientist\"\n",
    "\t* 16:  \"self-employed\"\n",
    "\t* 17:  \"technician/engineer\"\n",
    "\t* 18:  \"tradesman/craftsman\"\n",
    "\t* 19:  \"unemployed\"\n",
    "\t* 20:  \"writer\"\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>OccupationID</th>\n",
       "      <th>Zip-code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>55117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>7</td>\n",
       "      <td>02460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>20</td>\n",
       "      <td>55455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID Gender  Age  OccupationID Zip-code\n",
       "0       1      F    1            10    48067\n",
       "1       2      M   56            16    70072\n",
       "2       3      M   25            15    55117\n",
       "3       4      M   45             7    02460\n",
       "4       5      M   25            20    55455"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_title = ['UserID', 'Gender', 'Age', 'OccupationID', 'Zip-code']\n",
    "users = pd.read_table('./ml-1m/users.dat', sep='::', header=None, names=users_title, engine = 'python')\n",
    "users.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(users['OccupationID'].tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出UserID、Gender、Age和Occupation都是类别字段，其中邮编字段是我们不使用的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 电影数据\n",
    "分别有电影ID、电影名和电影风格等字段。\n",
    "\n",
    "数据中的格式：MovieID::Title::Genres\n",
    "\n",
    "- Titles are identical to titles provided by the IMDB (including\n",
    "year of release)\n",
    "- Genres are pipe-separated and are selected from the following genres:\n",
    "\n",
    "\t* Action\n",
    "\t* Adventure\n",
    "\t* Animation\n",
    "\t* Children's\n",
    "\t* Comedy\n",
    "\t* Crime\n",
    "\t* Documentary\n",
    "\t* Drama\n",
    "\t* Fantasy\n",
    "\t* Film-Noir\n",
    "\t* Horror\n",
    "\t* Musical\n",
    "\t* Mystery\n",
    "\t* Romance\n",
    "\t* Sci-Fi\n",
    "\t* Thriller\n",
    "\t* War\n",
    "\t* Western\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MovieID                               Title                        Genres\n",
       "0        1                    Toy Story (1995)   Animation|Children's|Comedy\n",
       "1        2                      Jumanji (1995)  Adventure|Children's|Fantasy\n",
       "2        3             Grumpier Old Men (1995)                Comedy|Romance\n",
       "3        4            Waiting to Exhale (1995)                  Comedy|Drama\n",
       "4        5  Father of the Bride Part II (1995)                        Comedy"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_title = ['MovieID', 'Title', 'Genres']\n",
    "movies = pd.read_table('./ml-1m/movies.dat', sep='::', header=None, names=movies_title, engine = 'python')\n",
    "movies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3883"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(movies['Title'].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MovieID是类别字段，Title是文本，Genres也是类别字段"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 评分数据\n",
    "分别有用户ID、电影ID、评分和时间戳等字段。\n",
    "\n",
    "数据中的格式：UserID::MovieID::Rating::Timestamp\n",
    "\n",
    "- UserIDs range between 1 and 6040 \n",
    "- MovieIDs range between 1 and 3952\n",
    "- Ratings are made on a 5-star scale (whole-star ratings only)\n",
    "- Timestamp is represented in seconds since the epoch as returned by time(2)\n",
    "- Each user has at least 20 ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "      <th>timestamps</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  MovieID  Rating  timestamps\n",
       "0       1     1193       5   978300760\n",
       "1       1      661       3   978302109\n",
       "2       1      914       3   978301968\n",
       "3       1     3408       4   978300275\n",
       "4       1     2355       5   978824291"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_title = ['UserID','MovieID', 'Rating', 'timestamps']\n",
    "ratings = pd.read_table('./ml-1m/ratings.dat', sep='::', header=None, names=ratings_title, engine = 'python')\n",
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "评分字段Rating就是我们要学习的targets，时间戳字段我们不使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000209, 4)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 来说说数据预处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- UserID、Occupation和MovieID不用变。\n",
    "- Gender字段：需要将‘F’和‘M’转换成0和1。\n",
    "- Age字段：要转成7个连续数字0~6。\n",
    "- Genres字段：是分类字段，要转成数字。首先将Genres中的类别转成字符串到数字的字典，然后再将每个电影的Genres字段转成数字列表，因为有些电影是多个Genres的组合。\n",
    "- Title字段：处理方式跟Genres字段一样，首先创建文本到数字的字典，然后将Title中的描述转成数字的列表。另外Title中的年份也需要去掉。\n",
    "- Genres和Title字段需要将长度统一，这样在神经网络中方便处理。空白部分用‘< PAD >’对应的数字填充。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实现数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data():\n",
    "    \"\"\"\n",
    "    Load Dataset from File\n",
    "    \"\"\"\n",
    "    #读取User数据\n",
    "    users_title = ['UserID', 'Gender', 'Age', 'JobID', 'Zip-code']\n",
    "    users = pd.read_table('./ml-1m/users.dat', sep='::', header=None, names=users_title, engine = 'python')\n",
    "    users = users.filter(regex='UserID|Gender|Age|JobID')\n",
    "    users_orig = users.values\n",
    "    #改变User数据中性别和年龄\n",
    "    gender_map = {'F':0, 'M':1}\n",
    "    users['Gender'] = users['Gender'].map(gender_map)\n",
    "\n",
    "    age_map = {val:ii for ii,val in enumerate(set(users['Age']))}\n",
    "    users['Age'] = users['Age'].map(age_map)\n",
    "\n",
    "    #读取Movie数据集\n",
    "    movies_title = ['MovieID', 'Title', 'Genres']\n",
    "    movies = pd.read_table('./ml-1m/movies.dat', sep='::', header=None, names=movies_title, engine = 'python')\n",
    "    movies_orig = movies.values\n",
    "    #将Title中的年份去掉\n",
    "    pattern = re.compile(r'^(.*)\\((\\d+)\\)$')\n",
    "\n",
    "    title_map = {val:pattern.match(val).group(1) for ii,val in enumerate(set(movies['Title']))}\n",
    "    movies['Title'] = movies['Title'].map(title_map)\n",
    "\n",
    "    #电影类型转数字字典\n",
    "    genres_set = set()\n",
    "    for val in movies['Genres'].str.split('|'):\n",
    "        genres_set.update(val)\n",
    "\n",
    "    genres_set.add('<PAD>')\n",
    "    genres2int = {val:ii for ii, val in enumerate(genres_set)}\n",
    "\n",
    "    #将电影类型转成等长数字列表，长度是18\n",
    "    genres_map = {val:[genres2int[row] for row in val.split('|')] for ii,val in enumerate(set(movies['Genres']))}\n",
    "\n",
    "    for key in genres_map:\n",
    "        for cnt in range(max(genres2int.values()) - len(genres_map[key])):\n",
    "            genres_map[key].insert(len(genres_map[key]) + cnt,genres2int['<PAD>'])\n",
    "    \n",
    "    movies['Genres'] = movies['Genres'].map(genres_map)\n",
    "\n",
    "    #电影Title转数字字典\n",
    "    title_set = set()\n",
    "    for val in movies['Title'].str.split():\n",
    "        title_set.update(val)\n",
    "    \n",
    "    title_set.add('<PAD>')\n",
    "    title2int = {val:ii for ii, val in enumerate(title_set)}\n",
    "\n",
    "    #将电影Title转成等长数字列表，长度是15\n",
    "    title_count = 15\n",
    "    title_map = {val:[title2int[row] for row in val.split()] for ii,val in enumerate(set(movies['Title']))}\n",
    "    \n",
    "    for key in title_map:\n",
    "        for cnt in range(title_count - len(title_map[key])):\n",
    "            title_map[key].insert(len(title_map[key]) + cnt,title2int['<PAD>'])\n",
    "    \n",
    "    movies['Title'] = movies['Title'].map(title_map)\n",
    "\n",
    "    #读取评分数据集\n",
    "    ratings_title = ['UserID','MovieID', 'ratings', 'timestamps']\n",
    "    ratings = pd.read_table('./ml-1m/ratings.dat', sep='::', header=None, names=ratings_title, engine = 'python')\n",
    "    ratings = ratings.filter(regex='UserID|MovieID|ratings')\n",
    "\n",
    "    #合并三个表\n",
    "    data = pd.merge(pd.merge(ratings, users), movies)\n",
    "    \n",
    "    #将数据分成X和y两张表\n",
    "    target_fields = ['ratings']\n",
    "    features_pd, targets_pd = data.drop(target_fields, axis=1), data[target_fields]\n",
    "    \n",
    "    features = features_pd.values\n",
    "    targets_values = targets_pd.values\n",
    "    \n",
    "    return title_count, title_set, genres2int, features, targets_values, ratings, users, movies, data, movies_orig, users_orig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 加载数据并保存到本地"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- title_count：Title字段的长度（15）\n",
    "- title_set：Title文本的集合\n",
    "- genres2int：电影类型转数字的字典\n",
    "- features：是输入X\n",
    "- targets_values：是学习目标y\n",
    "- ratings：评分数据集的Pandas对象\n",
    "- users：用户数据集的Pandas对象\n",
    "- movies：电影数据的Pandas对象\n",
    "- data：三个数据集组合在一起的Pandas对象\n",
    "- movies_orig：没有做数据处理的原始电影数据\n",
    "- users_orig：没有做数据处理的原始用户数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_count, title_set, genres2int, features, targets_values, ratings, users, movies, data, movies_orig, users_orig = load_data()\n",
    "\n",
    "\n",
    "pickle.dump((title_count, title_set, genres2int, features, targets_values, ratings, users, movies, data, movies_orig, users_orig), open('preprocess.p', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预处理后的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>JobID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  Gender  Age  JobID\n",
       "0       1       0    0     10\n",
       "1       2       1    5     16\n",
       "2       3       1    6     15\n",
       "3       4       1    2      7\n",
       "4       5       1    6     20"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>[5157, 4190, 2793, 2793, 2793, 2793, 2793, 279...</td>\n",
       "      <td>[2, 6, 12, 16, 16, 16, 16, 16, 16, 16, 16, 16,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>[3545, 2793, 2793, 2793, 2793, 2793, 2793, 279...</td>\n",
       "      <td>[7, 6, 0, 16, 16, 16, 16, 16, 16, 16, 16, 16, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>[4071, 2921, 4906, 2793, 2793, 2793, 2793, 279...</td>\n",
       "      <td>[12, 13, 16, 16, 16, 16, 16, 16, 16, 16, 16, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>[2047, 560, 1923, 2793, 2793, 2793, 2793, 2793...</td>\n",
       "      <td>[12, 11, 16, 16, 16, 16, 16, 16, 16, 16, 16, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>[2043, 924, 3678, 2765, 3826, 2556, 2793, 2793...</td>\n",
       "      <td>[12, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MovieID                                              Title  \\\n",
       "0        1  [5157, 4190, 2793, 2793, 2793, 2793, 2793, 279...   \n",
       "1        2  [3545, 2793, 2793, 2793, 2793, 2793, 2793, 279...   \n",
       "2        3  [4071, 2921, 4906, 2793, 2793, 2793, 2793, 279...   \n",
       "3        4  [2047, 560, 1923, 2793, 2793, 2793, 2793, 2793...   \n",
       "4        5  [2043, 924, 3678, 2765, 3826, 2556, 2793, 2793...   \n",
       "\n",
       "                                              Genres  \n",
       "0  [2, 6, 12, 16, 16, 16, 16, 16, 16, 16, 16, 16,...  \n",
       "1  [7, 6, 0, 16, 16, 16, 16, 16, 16, 16, 16, 16, ...  \n",
       "2  [12, 13, 16, 16, 16, 16, 16, 16, 16, 16, 16, 1...  \n",
       "3  [12, 11, 16, 16, 16, 16, 16, 16, 16, 16, 16, 1...  \n",
       "4  [12, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 1...  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1,\n",
       "       list([5157, 4190, 2793, 2793, 2793, 2793, 2793, 2793, 2793, 2793, 2793, 2793, 2793, 2793, 2793]),\n",
       "       list([2, 6, 12, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16])],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.values[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 从本地读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "title_count, title_set, genres2int, features, targets_values, ratings, users, movies, data, movies_orig, users_orig = pickle.load(open('preprocess.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型设计"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/model.001.jpeg\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过研究数据集中的字段类型，我们发现有一些是类别字段，通常的处理是将这些字段转成one hot编码，但是像UserID、MovieID这样的字段就会变成非常的稀疏，输入的维度急剧膨胀，这是我们不愿意见到的，毕竟我这小笔记本不像大厂动辄能处理数以亿计维度的输入：）\n",
    "\n",
    "所以在预处理数据时将这些字段转成了数字，我们用这个数字当做嵌入矩阵的索引，在网络的第一层使用了嵌入层，维度是（N，32）和（N，16）。\n",
    "\n",
    "电影类型的处理要多一步，有时一个电影有多个电影类型，这样从嵌入矩阵索引出来是一个（n，32）的矩阵，因为有多个类型嘛，我们要将这个矩阵求和，变成（1，32）的向量。\n",
    "\n",
    "电影名的处理比较特殊，没有使用循环神经网络，而是用了文本卷积网络，下文会进行说明。\n",
    "\n",
    "从嵌入层索引出特征以后，将各特征传入全连接层，将输出再次传入全连接层，最终分别得到（1，200）的用户特征和电影特征两个特征向量。\n",
    "\n",
    "我们的目的就是要训练出用户特征和电影特征，在实现推荐功能时使用。得到这两个特征以后，就可以选择任意的方式来拟合评分了。我使用了两种方式，一个是上图中画出的将两个特征做向量乘法，将结果与真实评分做回归，采用MSE优化损失。因为本质上这是一个回归问题，另一种方式是，将两个特征作为输入，再次传入全连接层，输出一个值，将输出值回归到真实评分，采用MSE优化损失。\n",
    "\n",
    "实际上第二个方式的MSE loss在0.8附近，第一个方式在1附近，5次迭代的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 文本卷积网络\n",
    "网络看起来像下面这样"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/text_cnn.png\"/>\n",
    "图片来自Kim Yoon的论文：[`Convolutional Neural Networks for Sentence Classification`](https://arxiv.org/abs/1408.5882)\n",
    "\n",
    "将卷积神经网络用于文本的文章建议你阅读[`Understanding Convolutional Neural Networks for NLP`](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "网络的第一层是词嵌入层，由每一个单词的嵌入向量组成的嵌入矩阵。下一层使用多个不同尺寸（窗口大小）的卷积核在嵌入矩阵上做卷积，窗口大小指的是每次卷积覆盖几个单词。这里跟对图像做卷积不太一样，图像的卷积通常用2x2、3x3、5x5之类的尺寸，而文本卷积要覆盖整个单词的嵌入向量，所以尺寸是（单词数，向量维度），比如每次滑动3个，4个或者5个单词。第三层网络是max pooling得到一个长向量，最后使用dropout做正则化，最终得到了电影Title的特征。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 辅助函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import os\n",
    "import pickle\n",
    "\n",
    "def save_params(params):\n",
    "    \"\"\"\n",
    "    Save parameters to file\n",
    "    \"\"\"\n",
    "    pickle.dump(params, open('params.p', 'wb'))\n",
    "\n",
    "\n",
    "def load_params():\n",
    "    \"\"\"\n",
    "    Load parameters from file\n",
    "    \"\"\"\n",
    "    return pickle.load(open('params.p', mode='rb'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 编码实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#嵌入矩阵的维度\n",
    "embed_dim = 32\n",
    "#用户ID个数\n",
    "uid_max = max(features.take(0,1)) + 1 # 6040\n",
    "#性别个数\n",
    "gender_max = max(features.take(2,1)) + 1 # 1 + 1 = 2\n",
    "#年龄类别个数\n",
    "age_max = max(features.take(3,1)) + 1 # 6 + 1 = 7\n",
    "#职业个数\n",
    "job_max = max(features.take(4,1)) + 1# 20 + 1 = 21\n",
    "\n",
    "#电影ID个数\n",
    "movie_id_max = max(features.take(1,1)) + 1 # 3952\n",
    "#电影类型个数\n",
    "movie_categories_max = max(genres2int.values()) + 1 # 18 + 1 = 19\n",
    "#电影名单词个数\n",
    "movie_title_max = len(title_set) # 5216\n",
    "\n",
    "#对电影类型嵌入向量做加和操作的标志，考虑过使用mean做平均，但是没实现mean\n",
    "combiner = \"sum\"\n",
    "\n",
    "#电影名长度\n",
    "sentences_size = title_count # = 15\n",
    "#文本卷积滑动窗口，分别滑动2, 3, 4, 5个单词\n",
    "window_sizes = {2, 3, 4, 5}\n",
    "#文本卷积核数量\n",
    "filter_num = 8\n",
    "\n",
    "#电影ID转下标的字典，数据集中电影ID跟下标不一致，比如第5行的数据电影ID不一定是5\n",
    "movieid2idx = {val[0]:i for i, val in enumerate(movies.values)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 超参"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of Epochs\n",
    "num_epochs = 1\n",
    "# Batch Size\n",
    "batch_size = 256\n",
    "\n",
    "dropout_keep = 0.5\n",
    "# Learning Rate\n",
    "learning_rate = 0.0001\n",
    "# Show stats for every n number of batches\n",
    "show_every_n_batches = 20\n",
    "\n",
    "save_dir = './save'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义输入的占位符"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_inputs():\n",
    "    uid = tf.placeholder(tf.int32, [None, 1], name=\"uid\")\n",
    "    user_gender = tf.placeholder(tf.int32, [None, 1], name=\"user_gender\")\n",
    "    user_age = tf.placeholder(tf.int32, [None, 1], name=\"user_age\")\n",
    "    user_job = tf.placeholder(tf.int32, [None, 1], name=\"user_job\")\n",
    "    \n",
    "    movie_id = tf.placeholder(tf.int32, [None, 1], name=\"movie_id\")\n",
    "    movie_categories = tf.placeholder(tf.int32, [None, 18], name=\"movie_categories\")\n",
    "    movie_titles = tf.placeholder(tf.int32, [None, 15], name=\"movie_titles\")\n",
    "    targets = tf.placeholder(tf.int32, [None, 1], name=\"targets\")\n",
    "    LearningRate = tf.placeholder(tf.float32, name = \"LearningRate\")\n",
    "    dropout_keep_prob = tf.placeholder(tf.float32, name = \"dropout_keep_prob\")\n",
    "    return uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, LearningRate, dropout_keep_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 定义User的嵌入矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_user_embedding(uid, user_gender, user_age, user_job):\n",
    "    with tf.name_scope(\"user_embedding\"):\n",
    "        uid_embed_matrix = tf.Variable(tf.random_uniform([uid_max, embed_dim], -1, 1), name = \"uid_embed_matrix\")\n",
    "        uid_embed_layer = tf.nn.embedding_lookup(uid_embed_matrix, uid, name = \"uid_embed_layer\")\n",
    "    \n",
    "        gender_embed_matrix = tf.Variable(tf.random_uniform([gender_max, embed_dim // 2], -1, 1), name= \"gender_embed_matrix\")\n",
    "        gender_embed_layer = tf.nn.embedding_lookup(gender_embed_matrix, user_gender, name = \"gender_embed_layer\")\n",
    "        \n",
    "        age_embed_matrix = tf.Variable(tf.random_uniform([age_max, embed_dim // 2], -1, 1), name=\"age_embed_matrix\")\n",
    "        age_embed_layer = tf.nn.embedding_lookup(age_embed_matrix, user_age, name=\"age_embed_layer\")\n",
    "        \n",
    "        job_embed_matrix = tf.Variable(tf.random_uniform([job_max, embed_dim // 2], -1, 1), name = \"job_embed_matrix\")\n",
    "        job_embed_layer = tf.nn.embedding_lookup(job_embed_matrix, user_job, name = \"job_embed_layer\")\n",
    "    return uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 将User的嵌入矩阵一起全连接生成User的特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_user_feature_layer(uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer):\n",
    "    with tf.name_scope(\"user_fc\"):\n",
    "        #第一层全连接\n",
    "        uid_fc_layer = tf.layers.dense(uid_embed_layer, embed_dim, name = \"uid_fc_layer\", activation=tf.nn.relu)\n",
    "        gender_fc_layer = tf.layers.dense(gender_embed_layer, embed_dim, name = \"gender_fc_layer\", activation=tf.nn.relu)\n",
    "        age_fc_layer = tf.layers.dense(age_embed_layer, embed_dim, name =\"age_fc_layer\", activation=tf.nn.relu)\n",
    "        job_fc_layer = tf.layers.dense(job_embed_layer, embed_dim, name = \"job_fc_layer\", activation=tf.nn.relu)\n",
    "        \n",
    "        #第二层全连接\n",
    "        user_combine_layer = tf.concat([uid_fc_layer, gender_fc_layer, age_fc_layer, job_fc_layer], 2)  #(?, 1, 128)\n",
    "        user_combine_layer = tf.contrib.layers.fully_connected(user_combine_layer, 200, tf.tanh)  #(?, 1, 200)\n",
    "    \n",
    "        user_combine_layer_flat = tf.reshape(user_combine_layer, [-1, 200])\n",
    "    return user_combine_layer, user_combine_layer_flat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 定义Movie ID的嵌入矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_id_embed_layer(movie_id):\n",
    "    with tf.name_scope(\"movie_embedding\"):\n",
    "        movie_id_embed_matrix = tf.Variable(tf.random_uniform([movie_id_max, embed_dim], -1, 1), name = \"movie_id_embed_matrix\")\n",
    "        movie_id_embed_layer = tf.nn.embedding_lookup(movie_id_embed_matrix, movie_id, name = \"movie_id_embed_layer\")\n",
    "    return movie_id_embed_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 对电影类型的多个嵌入向量做加和"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_categories_layers(movie_categories):\n",
    "    with tf.name_scope(\"movie_categories_layers\"):\n",
    "        movie_categories_embed_matrix = tf.Variable(tf.random_uniform([movie_categories_max, embed_dim], -1, 1), name = \"movie_categories_embed_matrix\")\n",
    "        movie_categories_embed_layer = tf.nn.embedding_lookup(movie_categories_embed_matrix, movie_categories, name = \"movie_categories_embed_layer\")\n",
    "        if combiner == \"sum\":\n",
    "            movie_categories_embed_layer = tf.reduce_sum(movie_categories_embed_layer, axis=1, keep_dims=True)\n",
    "    #     elif combiner == \"mean\":\n",
    "\n",
    "    return movie_categories_embed_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Movie Title的文本卷积网络实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_cnn_layer(movie_titles):\n",
    "    #从嵌入矩阵中得到电影名对应的各个单词的嵌入向量\n",
    "    with tf.name_scope(\"movie_embedding\"):\n",
    "        movie_title_embed_matrix = tf.Variable(tf.random_uniform([movie_title_max, embed_dim], -1, 1), name = \"movie_title_embed_matrix\")\n",
    "        movie_title_embed_layer = tf.nn.embedding_lookup(movie_title_embed_matrix, movie_titles, name = \"movie_title_embed_layer\")\n",
    "        movie_title_embed_layer_expand = tf.expand_dims(movie_title_embed_layer, -1)\n",
    "    \n",
    "    #对文本嵌入层使用不同尺寸的卷积核做卷积和最大池化\n",
    "    pool_layer_lst = []\n",
    "    for window_size in window_sizes:\n",
    "        with tf.name_scope(\"movie_txt_conv_maxpool_{}\".format(window_size)):\n",
    "            filter_weights = tf.Variable(tf.truncated_normal([window_size, embed_dim, 1, filter_num],stddev=0.1),name = \"filter_weights\")\n",
    "            filter_bias = tf.Variable(tf.constant(0.1, shape=[filter_num]), name=\"filter_bias\")\n",
    "            \n",
    "            conv_layer = tf.nn.conv2d(movie_title_embed_layer_expand, filter_weights, [1,1,1,1], padding=\"VALID\", name=\"conv_layer\")\n",
    "            relu_layer = tf.nn.relu(tf.nn.bias_add(conv_layer,filter_bias), name =\"relu_layer\")\n",
    "            \n",
    "            maxpool_layer = tf.nn.max_pool(relu_layer, [1,sentences_size - window_size + 1 ,1,1], [1,1,1,1], padding=\"VALID\", name=\"maxpool_layer\")\n",
    "            pool_layer_lst.append(maxpool_layer)\n",
    "\n",
    "    #Dropout层\n",
    "    with tf.name_scope(\"pool_dropout\"):\n",
    "        pool_layer = tf.concat(pool_layer_lst, 3, name =\"pool_layer\")\n",
    "        max_num = len(window_sizes) * filter_num\n",
    "        pool_layer_flat = tf.reshape(pool_layer , [-1, 1, max_num], name = \"pool_layer_flat\")\n",
    "    \n",
    "        dropout_layer = tf.nn.dropout(pool_layer_flat, dropout_keep_prob, name = \"dropout_layer\")\n",
    "    return pool_layer_flat, dropout_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 将Movie的各个层一起做全连接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_feature_layer(movie_id_embed_layer, movie_categories_embed_layer, dropout_layer):\n",
    "    with tf.name_scope(\"movie_fc\"):\n",
    "        #第一层全连接\n",
    "        movie_id_fc_layer = tf.layers.dense(movie_id_embed_layer, embed_dim, name = \"movie_id_fc_layer\", activation=tf.nn.relu)\n",
    "        movie_categories_fc_layer = tf.layers.dense(movie_categories_embed_layer, embed_dim, name = \"movie_categories_fc_layer\", activation=tf.nn.relu)\n",
    "    \n",
    "        #第二层全连接\n",
    "        movie_combine_layer = tf.concat([movie_id_fc_layer, movie_categories_fc_layer, dropout_layer], 2)  #(?, 1, 96)\n",
    "        movie_combine_layer = tf.contrib.layers.fully_connected(movie_combine_layer, 200, tf.tanh)  #(?, 1, 200)\n",
    "    \n",
    "        movie_combine_layer_flat = tf.reshape(movie_combine_layer, [-1, 200])\n",
    "    return movie_combine_layer, movie_combine_layer_flat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建计算图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-23-559a1ee9ce9e>:6: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "train_graph = tf.Graph()\n",
    "with train_graph.as_default():\n",
    "    #获取输入占位符\n",
    "    uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob = get_inputs()\n",
    "    #获取User的4个嵌入向量\n",
    "    uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer = get_user_embedding(uid, user_gender, user_age, user_job)\n",
    "    #得到用户特征\n",
    "    user_combine_layer, user_combine_layer_flat = get_user_feature_layer(uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer)\n",
    "    #获取电影ID的嵌入向量\n",
    "    movie_id_embed_layer = get_movie_id_embed_layer(movie_id)\n",
    "    #获取电影类型的嵌入向量\n",
    "    movie_categories_embed_layer = get_movie_categories_layers(movie_categories)\n",
    "    #获取电影名的特征向量\n",
    "    pool_layer_flat, dropout_layer = get_movie_cnn_layer(movie_titles)\n",
    "    #得到电影特征\n",
    "    movie_combine_layer, movie_combine_layer_flat = get_movie_feature_layer(movie_id_embed_layer, \n",
    "                                                                                movie_categories_embed_layer, \n",
    "                                                                                dropout_layer)\n",
    "    #计算出评分，要注意两个不同的方案，inference的名字（name值）是不一样的，后面做推荐时要根据name取得tensor\n",
    "    with tf.name_scope(\"inference\"):\n",
    "        #将用户特征和电影特征作为输入，经过全连接，输出一个值的方案\n",
    "#         inference_layer = tf.concat([user_combine_layer_flat, movie_combine_layer_flat], 1)  #(?, 200)\n",
    "#         inference = tf.layers.dense(inference_layer, 1,\n",
    "#                                     kernel_initializer=tf.truncated_normal_initializer(stddev=0.01), \n",
    "#                                     kernel_regularizer=tf.nn.l2_loss, name=\"inference\")\n",
    "        #简单的将用户特征和电影特征做矩阵乘法得到一个预测评分\n",
    "#        inference = tf.matmul(user_combine_layer_flat, tf.transpose(movie_combine_layer_flat))\n",
    "        inference = tf.reduce_sum(user_combine_layer_flat * movie_combine_layer_flat, axis=1)\n",
    "        inference = tf.expand_dims(inference, axis=1)\n",
    "\n",
    "    with tf.name_scope(\"loss\"):\n",
    "        # MSE损失，将计算值回归到评分\n",
    "        cost = tf.losses.mean_squared_error(targets, inference )\n",
    "        loss = tf.reduce_mean(cost)\n",
    "    # 优化损失 \n",
    "#     train_op = tf.train.AdamOptimizer(lr).minimize(loss)  #cost\n",
    "    global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n",
    "    optimizer = tf.train.AdamOptimizer(lr)\n",
    "    gradients = optimizer.compute_gradients(loss)  #cost\n",
    "    train_op = optimizer.apply_gradients(gradients, global_step=global_step)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'inference/ExpandDims:0' shape=(?, 1) dtype=float32>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 取得batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_batches(Xs, ys, batch_size):\n",
    "    for start in range(0, len(Xs), batch_size):\n",
    "        end = min(start + batch_size, len(Xs))\n",
    "        yield Xs[start:end], ys[start:end]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing to D:\\下载\\STUVincent_vincent-movie_recommender-master\\movie_recommender\\runs\\1598262830\n",
      "\n",
      "2020-08-24T17:54:00.675232: Epoch   0 Batch    0/3125   train_loss = 16.326\n",
      "2020-08-24T17:54:02.359233: Epoch   0 Batch   20/3125   train_loss = 5.091\n",
      "2020-08-24T17:54:04.007733: Epoch   0 Batch   40/3125   train_loss = 2.981\n",
      "2020-08-24T17:54:05.952732: Epoch   0 Batch   60/3125   train_loss = 2.731\n",
      "2020-08-24T17:54:07.627233: Epoch   0 Batch   80/3125   train_loss = 2.495\n",
      "2020-08-24T17:54:09.569731: Epoch   0 Batch  100/3125   train_loss = 2.256\n",
      "2020-08-24T17:54:11.839734: Epoch   0 Batch  120/3125   train_loss = 2.337\n",
      "2020-08-24T17:54:13.601257: Epoch   0 Batch  140/3125   train_loss = 2.068\n",
      "2020-08-24T17:54:15.571258: Epoch   0 Batch  160/3125   train_loss = 1.692\n",
      "2020-08-24T17:54:17.366258: Epoch   0 Batch  180/3125   train_loss = 1.701\n",
      "2020-08-24T17:54:19.089291: Epoch   0 Batch  200/3125   train_loss = 1.830\n",
      "2020-08-24T17:54:21.160291: Epoch   0 Batch  220/3125   train_loss = 1.803\n",
      "2020-08-24T17:54:22.984291: Epoch   0 Batch  240/3125   train_loss = 1.693\n",
      "2020-08-24T17:54:24.743270: Epoch   0 Batch  260/3125   train_loss = 1.617\n",
      "2020-08-24T17:54:26.658291: Epoch   0 Batch  280/3125   train_loss = 1.632\n",
      "2020-08-24T17:54:28.346772: Epoch   0 Batch  300/3125   train_loss = 1.659\n",
      "2020-08-24T17:54:30.140790: Epoch   0 Batch  320/3125   train_loss = 1.618\n",
      "2020-08-24T17:54:32.193269: Epoch   0 Batch  340/3125   train_loss = 1.389\n",
      "2020-08-24T17:54:34.108293: Epoch   0 Batch  360/3125   train_loss = 1.573\n",
      "2020-08-24T17:54:36.146291: Epoch   0 Batch  380/3125   train_loss = 1.419\n",
      "2020-08-24T17:54:38.022968: Epoch   0 Batch  400/3125   train_loss = 1.250\n",
      "2020-08-24T17:54:39.916468: Epoch   0 Batch  420/3125   train_loss = 1.310\n",
      "2020-08-24T17:54:41.733989: Epoch   0 Batch  440/3125   train_loss = 1.590\n",
      "2020-08-24T17:54:43.698992: Epoch   0 Batch  460/3125   train_loss = 1.512\n",
      "2020-08-24T17:54:45.307491: Epoch   0 Batch  480/3125   train_loss = 1.421\n",
      "2020-08-24T17:54:46.961973: Epoch   0 Batch  500/3125   train_loss = 1.118\n",
      "2020-08-24T17:54:48.974473: Epoch   0 Batch  520/3125   train_loss = 1.447\n",
      "2020-08-24T17:54:50.698525: Epoch   0 Batch  540/3125   train_loss = 1.272\n",
      "2020-08-24T17:54:52.270544: Epoch   0 Batch  560/3125   train_loss = 1.445\n",
      "2020-08-24T17:54:54.141524: Epoch   0 Batch  580/3125   train_loss = 1.408\n",
      "2020-08-24T17:54:56.066544: Epoch   0 Batch  600/3125   train_loss = 1.447\n",
      "2020-08-24T17:54:57.659045: Epoch   0 Batch  620/3125   train_loss = 1.420\n",
      "2020-08-24T17:54:59.353543: Epoch   0 Batch  640/3125   train_loss = 1.416\n",
      "2020-08-24T17:55:01.351532: Epoch   0 Batch  660/3125   train_loss = 1.372\n",
      "2020-08-24T17:55:03.054073: Epoch   0 Batch  680/3125   train_loss = 1.260\n",
      "2020-08-24T17:55:04.663574: Epoch   0 Batch  700/3125   train_loss = 1.338\n",
      "2020-08-24T17:55:06.638553: Epoch   0 Batch  720/3125   train_loss = 1.316\n",
      "2020-08-24T17:55:08.472574: Epoch   0 Batch  740/3125   train_loss = 1.360\n",
      "2020-08-24T17:55:10.053059: Epoch   0 Batch  760/3125   train_loss = 1.430\n",
      "2020-08-24T17:55:11.826557: Epoch   0 Batch  780/3125   train_loss = 1.425\n",
      "2020-08-24T17:55:13.837573: Epoch   0 Batch  800/3125   train_loss = 1.299\n",
      "2020-08-24T17:55:15.588075: Epoch   0 Batch  820/3125   train_loss = 1.307\n",
      "2020-08-24T17:55:17.231573: Epoch   0 Batch  840/3125   train_loss = 1.315\n",
      "2020-08-24T17:55:19.292574: Epoch   0 Batch  860/3125   train_loss = 1.271\n",
      "2020-08-24T17:55:21.040576: Epoch   0 Batch  880/3125   train_loss = 1.266\n",
      "2020-08-24T17:55:22.626575: Epoch   0 Batch  900/3125   train_loss = 1.226\n",
      "2020-08-24T17:55:24.487556: Epoch   0 Batch  920/3125   train_loss = 1.300\n",
      "2020-08-24T17:55:26.400076: Epoch   0 Batch  940/3125   train_loss = 1.411\n",
      "2020-08-24T17:55:28.006076: Epoch   0 Batch  960/3125   train_loss = 1.344\n",
      "2020-08-24T17:55:29.784559: Epoch   0 Batch  980/3125   train_loss = 1.324\n",
      "2020-08-24T17:55:31.819571: Epoch   0 Batch 1000/3125   train_loss = 1.287\n",
      "2020-08-24T17:55:33.500059: Epoch   0 Batch 1020/3125   train_loss = 1.458\n",
      "2020-08-24T17:55:35.076576: Epoch   0 Batch 1040/3125   train_loss = 1.208\n",
      "2020-08-24T17:55:37.037576: Epoch   0 Batch 1060/3125   train_loss = 1.586\n",
      "2020-08-24T17:55:38.903577: Epoch   0 Batch 1080/3125   train_loss = 1.199\n",
      "2020-08-24T17:55:40.489577: Epoch   0 Batch 1100/3125   train_loss = 1.344\n",
      "2020-08-24T17:55:42.269576: Epoch   0 Batch 1120/3125   train_loss = 1.275\n",
      "2020-08-24T17:55:44.269576: Epoch   0 Batch 1140/3125   train_loss = 1.334\n",
      "2020-08-24T17:55:45.884578: Epoch   0 Batch 1160/3125   train_loss = 1.204\n",
      "2020-08-24T17:55:47.455077: Epoch   0 Batch 1180/3125   train_loss = 1.297\n",
      "2020-08-24T17:55:49.444078: Epoch   0 Batch 1200/3125   train_loss = 1.316\n",
      "2020-08-24T17:55:51.232078: Epoch   0 Batch 1220/3125   train_loss = 1.125\n",
      "2020-08-24T17:55:52.825578: Epoch   0 Batch 1240/3125   train_loss = 1.187\n",
      "2020-08-24T17:55:54.626077: Epoch   0 Batch 1260/3125   train_loss = 1.225\n",
      "2020-08-24T17:55:56.567580: Epoch   0 Batch 1280/3125   train_loss = 1.258\n",
      "2020-08-24T17:55:58.214559: Epoch   0 Batch 1300/3125   train_loss = 1.275\n",
      "2020-08-24T17:55:59.900060: Epoch   0 Batch 1320/3125   train_loss = 1.174\n",
      "2020-08-24T17:56:01.918576: Epoch   0 Batch 1340/3125   train_loss = 1.085\n",
      "2020-08-24T17:56:03.626083: Epoch   0 Batch 1360/3125   train_loss = 1.227\n",
      "2020-08-24T17:56:05.195580: Epoch   0 Batch 1380/3125   train_loss = 1.114\n",
      "2020-08-24T17:56:07.079579: Epoch   0 Batch 1400/3125   train_loss = 1.296\n",
      "2020-08-24T17:56:08.981580: Epoch   0 Batch 1420/3125   train_loss = 1.376\n",
      "2020-08-24T17:56:10.564558: Epoch   0 Batch 1440/3125   train_loss = 1.104\n",
      "2020-08-24T17:56:12.310580: Epoch   0 Batch 1460/3125   train_loss = 1.219\n",
      "2020-08-24T17:56:14.293115: Epoch   0 Batch 1480/3125   train_loss = 1.220\n",
      "2020-08-24T17:56:15.965618: Epoch   0 Batch 1500/3125   train_loss = 1.374\n",
      "2020-08-24T17:56:17.612118: Epoch   0 Batch 1520/3125   train_loss = 1.286\n",
      "2020-08-24T17:56:19.554116: Epoch   0 Batch 1540/3125   train_loss = 1.278\n",
      "2020-08-24T17:56:21.360109: Epoch   0 Batch 1560/3125   train_loss = 1.261\n",
      "2020-08-24T17:56:22.981596: Epoch   0 Batch 1580/3125   train_loss = 1.216\n",
      "2020-08-24T17:56:24.816616: Epoch   0 Batch 1600/3125   train_loss = 1.361\n",
      "2020-08-24T17:56:26.773099: Epoch   0 Batch 1620/3125   train_loss = 1.181\n",
      "2020-08-24T17:56:28.403598: Epoch   0 Batch 1640/3125   train_loss = 1.343\n",
      "2020-08-24T17:56:30.029117: Epoch   0 Batch 1660/3125   train_loss = 1.320\n",
      "2020-08-24T17:56:32.012651: Epoch   0 Batch 1680/3125   train_loss = 1.239\n",
      "2020-08-24T17:56:33.803151: Epoch   0 Batch 1700/3125   train_loss = 1.071\n",
      "2020-08-24T17:56:35.393701: Epoch   0 Batch 1720/3125   train_loss = 1.217\n",
      "2020-08-24T17:56:37.373700: Epoch   0 Batch 1740/3125   train_loss = 1.167\n",
      "2020-08-24T17:56:39.291202: Epoch   0 Batch 1760/3125   train_loss = 1.336\n",
      "2020-08-24T17:56:40.889682: Epoch   0 Batch 1780/3125   train_loss = 1.206\n",
      "2020-08-24T17:56:42.835701: Epoch   0 Batch 1800/3125   train_loss = 1.214\n",
      "2020-08-24T17:56:44.692702: Epoch   0 Batch 1820/3125   train_loss = 1.242\n",
      "2020-08-24T17:56:46.352201: Epoch   0 Batch 1840/3125   train_loss = 1.236\n",
      "2020-08-24T17:56:48.349702: Epoch   0 Batch 1860/3125   train_loss = 1.269\n",
      "2020-08-24T17:56:50.086702: Epoch   0 Batch 1880/3125   train_loss = 1.294\n",
      "2020-08-24T17:56:51.854702: Epoch   0 Batch 1900/3125   train_loss = 1.059\n",
      "2020-08-24T17:56:53.984183: Epoch   0 Batch 1920/3125   train_loss = 1.208\n",
      "2020-08-24T17:56:55.700784: Epoch   0 Batch 1940/3125   train_loss = 1.174\n",
      "2020-08-24T17:56:57.598284: Epoch   0 Batch 1960/3125   train_loss = 1.146\n",
      "2020-08-24T17:56:59.524285: Epoch   0 Batch 1980/3125   train_loss = 1.124\n",
      "2020-08-24T17:57:01.095285: Epoch   0 Batch 2000/3125   train_loss = 1.420\n",
      "2020-08-24T17:57:03.065336: Epoch   0 Batch 2020/3125   train_loss = 1.274\n",
      "2020-08-24T17:57:04.951356: Epoch   0 Batch 2040/3125   train_loss = 1.190\n",
      "2020-08-24T17:57:06.655332: Epoch   0 Batch 2060/3125   train_loss = 1.090\n",
      "2020-08-24T17:57:08.606354: Epoch   0 Batch 2080/3125   train_loss = 1.319\n",
      "2020-08-24T17:57:11.285336: Epoch   0 Batch 2100/3125   train_loss = 1.103\n",
      "2020-08-24T17:57:13.871132: Epoch   0 Batch 2120/3125   train_loss = 1.074\n",
      "2020-08-24T17:57:16.029134: Epoch   0 Batch 2140/3125   train_loss = 1.158\n",
      "2020-08-24T17:57:17.943133: Epoch   0 Batch 2160/3125   train_loss = 1.133\n",
      "2020-08-24T17:57:20.148172: Epoch   0 Batch 2180/3125   train_loss = 1.136\n",
      "2020-08-24T17:57:21.951148: Epoch   0 Batch 2200/3125   train_loss = 1.172\n",
      "2020-08-24T17:57:24.027672: Epoch   0 Batch 2220/3125   train_loss = 1.159\n",
      "2020-08-24T17:57:26.084927: Epoch   0 Batch 2240/3125   train_loss = 1.016\n",
      "2020-08-24T17:57:28.002448: Epoch   0 Batch 2260/3125   train_loss = 1.132\n",
      "2020-08-24T17:57:30.584428: Epoch   0 Batch 2280/3125   train_loss = 1.224\n",
      "2020-08-24T17:57:32.257951: Epoch   0 Batch 2300/3125   train_loss = 1.329\n",
      "2020-08-24T17:57:34.201948: Epoch   0 Batch 2320/3125   train_loss = 1.325\n",
      "2020-08-24T17:57:36.229451: Epoch   0 Batch 2340/3125   train_loss = 1.242\n",
      "2020-08-24T17:57:38.047944: Epoch   0 Batch 2360/3125   train_loss = 1.242\n",
      "2020-08-24T17:57:40.212444: Epoch   0 Batch 2380/3125   train_loss = 1.143\n",
      "2020-08-24T17:57:42.619427: Epoch   0 Batch 2400/3125   train_loss = 1.272\n",
      "2020-08-24T17:57:45.532928: Epoch   0 Batch 2420/3125   train_loss = 1.126\n",
      "2020-08-24T17:57:47.262947: Epoch   0 Batch 2440/3125   train_loss = 1.253\n",
      "2020-08-24T17:57:49.239932: Epoch   0 Batch 2460/3125   train_loss = 1.135\n",
      "2020-08-24T17:57:52.053976: Epoch   0 Batch 2480/3125   train_loss = 1.251\n",
      "2020-08-24T17:57:54.512994: Epoch   0 Batch 2500/3125   train_loss = 1.235\n",
      "2020-08-24T17:57:57.044998: Epoch   0 Batch 2520/3125   train_loss = 1.127\n",
      "2020-08-24T17:57:59.431977: Epoch   0 Batch 2540/3125   train_loss = 1.042\n",
      "2020-08-24T17:58:01.892974: Epoch   0 Batch 2560/3125   train_loss = 1.014\n",
      "2020-08-24T17:58:04.659474: Epoch   0 Batch 2580/3125   train_loss = 1.181\n",
      "2020-08-24T17:58:06.780474: Epoch   0 Batch 2600/3125   train_loss = 1.252\n",
      "2020-08-24T17:58:08.692478: Epoch   0 Batch 2620/3125   train_loss = 1.079\n",
      "2020-08-24T17:58:12.075497: Epoch   0 Batch 2640/3125   train_loss = 1.180\n",
      "2020-08-24T17:58:14.522475: Epoch   0 Batch 2660/3125   train_loss = 1.238\n",
      "2020-08-24T17:58:16.619497: Epoch   0 Batch 2680/3125   train_loss = 1.108\n",
      "2020-08-24T17:58:19.307479: Epoch   0 Batch 2700/3125   train_loss = 1.238\n",
      "2020-08-24T17:58:21.638476: Epoch   0 Batch 2720/3125   train_loss = 1.121\n",
      "2020-08-24T17:58:24.321979: Epoch   0 Batch 2740/3125   train_loss = 1.211\n",
      "2020-08-24T17:58:26.486553: Epoch   0 Batch 2760/3125   train_loss = 1.235\n",
      "2020-08-24T17:58:28.212040: Epoch   0 Batch 2780/3125   train_loss = 1.111\n",
      "2020-08-24T17:58:30.320532: Epoch   0 Batch 2800/3125   train_loss = 1.339\n",
      "2020-08-24T17:58:32.282119: Epoch   0 Batch 2820/3125   train_loss = 1.369\n",
      "2020-08-24T17:58:34.207140: Epoch   0 Batch 2840/3125   train_loss = 1.140\n",
      "2020-08-24T17:58:36.389663: Epoch   0 Batch 2860/3125   train_loss = 1.100\n",
      "2020-08-24T17:58:38.327142: Epoch   0 Batch 2880/3125   train_loss = 1.185\n",
      "2020-08-24T17:58:41.651646: Epoch   0 Batch 2900/3125   train_loss = 1.162\n",
      "2020-08-24T17:58:43.474162: Epoch   0 Batch 2920/3125   train_loss = 1.106\n",
      "2020-08-24T17:58:45.634141: Epoch   0 Batch 2940/3125   train_loss = 1.160\n",
      "2020-08-24T17:58:47.734165: Epoch   0 Batch 2960/3125   train_loss = 1.194\n",
      "2020-08-24T17:58:50.440163: Epoch   0 Batch 2980/3125   train_loss = 1.155\n",
      "2020-08-24T17:58:52.917178: Epoch   0 Batch 3000/3125   train_loss = 1.143\n",
      "2020-08-24T17:58:55.330196: Epoch   0 Batch 3020/3125   train_loss = 1.179\n",
      "2020-08-24T17:58:57.654199: Epoch   0 Batch 3040/3125   train_loss = 1.102\n",
      "2020-08-24T17:58:59.328194: Epoch   0 Batch 3060/3125   train_loss = 1.122\n",
      "2020-08-24T17:59:01.433698: Epoch   0 Batch 3080/3125   train_loss = 1.211\n",
      "2020-08-24T17:59:03.554198: Epoch   0 Batch 3100/3125   train_loss = 1.184\n",
      "2020-08-24T17:59:05.733198: Epoch   0 Batch 3120/3125   train_loss = 1.081\n",
      "2020-08-24T17:59:06.848679: Epoch   0 Batch    0/781   test_loss = 1.019\n",
      "2020-08-24T17:59:07.286182: Epoch   0 Batch   20/781   test_loss = 1.139\n",
      "2020-08-24T17:59:07.728680: Epoch   0 Batch   40/781   test_loss = 1.134\n",
      "2020-08-24T17:59:08.061678: Epoch   0 Batch   60/781   test_loss = 1.341\n",
      "2020-08-24T17:59:08.381678: Epoch   0 Batch   80/781   test_loss = 1.447\n",
      "2020-08-24T17:59:08.847181: Epoch   0 Batch  100/781   test_loss = 1.355\n",
      "2020-08-24T17:59:09.202180: Epoch   0 Batch  120/781   test_loss = 1.233\n",
      "2020-08-24T17:59:09.515679: Epoch   0 Batch  140/781   test_loss = 1.258\n",
      "2020-08-24T17:59:10.117683: Epoch   0 Batch  160/781   test_loss = 1.402\n",
      "2020-08-24T17:59:11.148679: Epoch   0 Batch  180/781   test_loss = 1.302\n",
      "2020-08-24T17:59:11.824179: Epoch   0 Batch  200/781   test_loss = 1.189\n",
      "2020-08-24T17:59:12.319184: Epoch   0 Batch  220/781   test_loss = 0.991\n",
      "2020-08-24T17:59:12.824683: Epoch   0 Batch  240/781   test_loss = 1.216\n",
      "2020-08-24T17:59:13.574681: Epoch   0 Batch  260/781   test_loss = 1.248\n",
      "2020-08-24T17:59:14.032681: Epoch   0 Batch  280/781   test_loss = 1.442\n",
      "2020-08-24T17:59:14.476182: Epoch   0 Batch  300/781   test_loss = 1.277\n",
      "2020-08-24T17:59:14.975684: Epoch   0 Batch  320/781   test_loss = 1.371\n",
      "2020-08-24T17:59:15.402181: Epoch   0 Batch  340/781   test_loss = 0.938\n",
      "2020-08-24T17:59:15.889182: Epoch   0 Batch  360/781   test_loss = 1.315\n",
      "2020-08-24T17:59:16.633178: Epoch   0 Batch  380/781   test_loss = 1.219\n",
      "2020-08-24T17:59:17.097182: Epoch   0 Batch  400/781   test_loss = 1.149\n",
      "2020-08-24T17:59:17.564682: Epoch   0 Batch  420/781   test_loss = 1.062\n",
      "2020-08-24T17:59:18.003182: Epoch   0 Batch  440/781   test_loss = 1.267\n",
      "2020-08-24T17:59:18.542680: Epoch   0 Batch  460/781   test_loss = 1.077\n",
      "2020-08-24T17:59:19.015680: Epoch   0 Batch  480/781   test_loss = 1.158\n",
      "2020-08-24T17:59:19.974680: Epoch   0 Batch  500/781   test_loss = 1.031\n",
      "2020-08-24T17:59:20.359179: Epoch   0 Batch  520/781   test_loss = 1.228\n",
      "2020-08-24T17:59:20.872182: Epoch   0 Batch  540/781   test_loss = 1.024\n",
      "2020-08-24T17:59:21.298680: Epoch   0 Batch  560/781   test_loss = 1.339\n",
      "2020-08-24T17:59:21.669682: Epoch   0 Batch  580/781   test_loss = 1.159\n",
      "2020-08-24T17:59:22.042677: Epoch   0 Batch  600/781   test_loss = 1.208\n",
      "2020-08-24T17:59:22.429680: Epoch   0 Batch  620/781   test_loss = 1.266\n",
      "2020-08-24T17:59:22.878180: Epoch   0 Batch  640/781   test_loss = 1.383\n",
      "2020-08-24T17:59:23.245180: Epoch   0 Batch  660/781   test_loss = 1.124\n",
      "2020-08-24T17:59:23.663681: Epoch   0 Batch  680/781   test_loss = 1.404\n",
      "2020-08-24T17:59:24.007683: Epoch   0 Batch  700/781   test_loss = 1.148\n",
      "2020-08-24T17:59:24.360179: Epoch   0 Batch  720/781   test_loss = 1.318\n",
      "2020-08-24T17:59:24.729183: Epoch   0 Batch  740/781   test_loss = 1.240\n",
      "2020-08-24T17:59:25.095181: Epoch   0 Batch  760/781   test_loss = 1.208\n",
      "2020-08-24T17:59:25.509681: Epoch   0 Batch  780/781   test_loss = 1.178\n",
      "Model Trained and Saved\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "import datetime\n",
    "\n",
    "losses = {'train':[], 'test':[]}\n",
    "\n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    \n",
    "    #搜集数据给tensorBoard用\n",
    "    # Keep track of gradient values and sparsity\n",
    "    grad_summaries = []\n",
    "    for g, v in gradients:\n",
    "        if g is not None:\n",
    "            grad_hist_summary = tf.summary.histogram(\"{}/grad/hist\".format(v.name.replace(':', '_')), g)\n",
    "            sparsity_summary = tf.summary.scalar(\"{}/grad/sparsity\".format(v.name.replace(':', '_')), tf.nn.zero_fraction(g))\n",
    "            grad_summaries.append(grad_hist_summary)\n",
    "            grad_summaries.append(sparsity_summary)\n",
    "    grad_summaries_merged = tf.summary.merge(grad_summaries)\n",
    "        \n",
    "    # Output directory for models and summaries\n",
    "    timestamp = str(int(time.time()))\n",
    "    out_dir = os.path.abspath(os.path.join(os.path.curdir, \"runs\", timestamp))\n",
    "    print(\"Writing to {}\\n\".format(out_dir))\n",
    "     \n",
    "    # Summaries for loss and accuracy\n",
    "    loss_summary = tf.summary.scalar(\"loss\", loss)\n",
    "\n",
    "    # Train Summaries\n",
    "    train_summary_op = tf.summary.merge([loss_summary, grad_summaries_merged])\n",
    "    train_summary_dir = os.path.join(out_dir, \"summaries\", \"train\")\n",
    "    train_summary_writer = tf.summary.FileWriter(train_summary_dir, sess.graph)\n",
    "\n",
    "    # Inference summaries\n",
    "    inference_summary_op = tf.summary.merge([loss_summary])\n",
    "    inference_summary_dir = os.path.join(out_dir, \"summaries\", \"inference\")\n",
    "    inference_summary_writer = tf.summary.FileWriter(inference_summary_dir, sess.graph)\n",
    "\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    saver = tf.train.Saver()\n",
    "    for epoch_i in range(num_epochs):\n",
    "        \n",
    "        #将数据集分成训练集和测试集，随机种子不固定\n",
    "        train_X,test_X, train_y, test_y = train_test_split(features,  \n",
    "                                                           targets_values,  \n",
    "                                                           test_size = 0.2,  \n",
    "                                                           random_state = 0)  \n",
    "        \n",
    "        train_batches = get_batches(train_X, train_y, batch_size)\n",
    "        test_batches = get_batches(test_X, test_y, batch_size)\n",
    "    \n",
    "        #训练的迭代，保存训练损失\n",
    "        for batch_i in range(len(train_X) // batch_size):\n",
    "            x, y = next(train_batches)\n",
    "\n",
    "            categories = np.zeros([batch_size, 18])\n",
    "            for i in range(batch_size):\n",
    "                categories[i] = x.take(6,1)[i]\n",
    "\n",
    "            titles = np.zeros([batch_size, sentences_size])\n",
    "            for i in range(batch_size):\n",
    "                titles[i] = x.take(5,1)[i]\n",
    "\n",
    "            feed = {\n",
    "                uid: np.reshape(x.take(0,1), [batch_size, 1]),\n",
    "                user_gender: np.reshape(x.take(2,1), [batch_size, 1]),\n",
    "                user_age: np.reshape(x.take(3,1), [batch_size, 1]),\n",
    "                user_job: np.reshape(x.take(4,1), [batch_size, 1]),\n",
    "                movie_id: np.reshape(x.take(1,1), [batch_size, 1]),\n",
    "                movie_categories: categories,  #x.take(6,1)\n",
    "                movie_titles: titles,  #x.take(5,1)\n",
    "                targets: np.reshape(y, [batch_size, 1]),\n",
    "                dropout_keep_prob: dropout_keep, #dropout_keep\n",
    "                lr: learning_rate}\n",
    "\n",
    "            step, train_loss, summaries, _ = sess.run([global_step, loss, train_summary_op, train_op], feed)  #cost\n",
    "            losses['train'].append(train_loss)\n",
    "            train_summary_writer.add_summary(summaries, step)  #\n",
    "            \n",
    "            # Show every <show_every_n_batches> batches\n",
    "            if (epoch_i * (len(train_X) // batch_size) + batch_i) % show_every_n_batches == 0:\n",
    "                time_str = datetime.datetime.now().isoformat()\n",
    "                print('{}: Epoch {:>3} Batch {:>4}/{}   train_loss = {:.3f}'.format(\n",
    "                    time_str,\n",
    "                    epoch_i,\n",
    "                    batch_i,\n",
    "                    (len(train_X) // batch_size),\n",
    "                    train_loss))\n",
    "                \n",
    "        #使用测试数据的迭代\n",
    "        for batch_i  in range(len(test_X) // batch_size):\n",
    "            x, y = next(test_batches)\n",
    "            \n",
    "            categories = np.zeros([batch_size, 18])\n",
    "            for i in range(batch_size):\n",
    "                categories[i] = x.take(6,1)[i]\n",
    "\n",
    "            titles = np.zeros([batch_size, sentences_size])\n",
    "            for i in range(batch_size):\n",
    "                titles[i] = x.take(5,1)[i]\n",
    "\n",
    "            feed = {\n",
    "                uid: np.reshape(x.take(0,1), [batch_size, 1]),\n",
    "                user_gender: np.reshape(x.take(2,1), [batch_size, 1]),\n",
    "                user_age: np.reshape(x.take(3,1), [batch_size, 1]),\n",
    "                user_job: np.reshape(x.take(4,1), [batch_size, 1]),\n",
    "                movie_id: np.reshape(x.take(1,1), [batch_size, 1]),\n",
    "                movie_categories: categories,  #x.take(6,1)\n",
    "                movie_titles: titles,  #x.take(5,1)\n",
    "                targets: np.reshape(y, [batch_size, 1]),\n",
    "                dropout_keep_prob: 1,\n",
    "                lr: learning_rate}\n",
    "            \n",
    "            step, test_loss, summaries = sess.run([global_step, loss, inference_summary_op], feed)  #cost\n",
    "\n",
    "            #保存测试损失\n",
    "            losses['test'].append(test_loss)\n",
    "            inference_summary_writer.add_summary(summaries, step)  #\n",
    "\n",
    "            time_str = datetime.datetime.now().isoformat()\n",
    "            if (epoch_i * (len(test_X) // batch_size) + batch_i) % show_every_n_batches == 0:\n",
    "                print('{}: Epoch {:>3} Batch {:>4}/{}   test_loss = {:.3f}'.format(\n",
    "                    time_str,\n",
    "                    epoch_i,\n",
    "                    batch_i,\n",
    "                    (len(test_X) // batch_size),\n",
    "                    test_loss))\n",
    "\n",
    "    # Save Model\n",
    "    saver.save(sess, save_dir)  #, global_step=epoch_i\n",
    "    print('Model Trained and Saved')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 在 TensorBoard 中查看可视化结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tensorboard --logdir /PATH_TO_CODE/runs/1513402825/summaries/\n",
    "\n",
    "<img src=\"assets/loss.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存参数\n",
    "保存`save_dir` 在生成预测时使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'save_dir' is not defined",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-30-88ebbb2a383e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0msave_params\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msave_dir\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mload_dir\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mload_params\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'save_dir' is not defined"
     ],
     "output_type": "error"
    }
   ],
   "source": [
    "save_params((save_dir))\n",
    "\n",
    "load_dir = load_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 显示训练Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvEAAAHwCAYAAAAvjDDZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdeXhV1b3/8c86mRMyEpIwzwYEEQFFUQERBGcraO21rUOdalvFagdrrVQ7oL29Tq3Dvc76qziDI4oiKIOCYRJBZTCEKYQQyDyf9fsjJzEhiQSz9znZ4f16njw7Z+3pCyT6Oeustbax1goAAACAd/hCXQAAAACAw0OIBwAAADyGEA8AAAB4DCEeAAAA8BhCPAAAAOAxhHgAAADAYwjxAAAAgMcQ4gEAAACPIcQDAAAAHkOIBwAAADyGEA8AAAB4DCEeAAAA8JjwUBfgBmPMN5ISJGWHuBQAAAB0bv0kFVlr+wfzpp0yxEtKiImJSRk6dGhKqAsBAABA57Vx40aVl5cH/b6dNcRnDx06NCUrKyvUdQAAAKATGz16tFatWpUd7PsyJh4AAADwGEI8AAAA4DGEeAAAAMBjCPEAAACAxxDiAQAAAI8hxAMAAAAeQ4gHAAAAPKazrhMPAAAOk9/vV0FBgYqLi1VZWSlrbahLAoLKGKOoqCjFx8crJSVFPl/H7e8mxAMAAPn9fm3fvl1lZWWhLgUIGWutKioqVFFRodLSUvXu3bvDBnlCPAAAUEFBgcrKyhQeHq6MjAzFxcV12PACuMXv96u0tFS5ubkqKytTQUGBUlNTQ11Wixz57TTG3G2M+cAYs90YU26MKTDGrDbG3GGM6XrQsf2MMfY7vuY4URMAAGi74uJiSVJGRobi4+MJ8Dgi+Xw+xcfHKyMjQ9K3vxcdkVM98TdJWiVpgaQ8SXGSTpQ0S9I1xpgTrbXbDzpnraS5LVxrvUM1AQCANqqsrJQkxcXFhbgSIPTqfw/qfy86IqdCfIK1tuLgRmPMXyX9QdKtkq4/aPcaa+0sh+4PAADaoX4SKz3wQN0EV0kdenK3I7+pLQX4gBcD28FO3AcAAABwW32I78jcnth6bmC7roV9PYwx10rqKmmfpOXW2paOa5UxJquVXUMO5zoAAACAlzga4o0xt0jqIilR0hhJp6guwM9u4fApga/G5y+SdJm1NsfJuoLJWuuJd28AAADwLqcHvt0i6Q5JM1UX4OdLOsNau7fRMWWS7pI0WlJy4GuCpA8lTZT0gTGmTbNqrLWjW/qS9KVTf6DD8ec3vtDxf/1Ar63eEYrbAwAADzPGaOLEie2+zsSJEztch+JTTz0lY4yeeuqpUJfSaTga4q21GdZaIylD0oWSBkhabYwZ1eiYPGvtn6y1q6y1BwJfH0k6Q9KnkgZJusrJuoJhy94SPbk0W/kllbrphbWhLgcAABwmY8xhfRFIEUqujIm31u6R9JoxZpWkryU9I2n4Ic6pMcY8JmmspPGS7nejNrfsKWxtbi8AAPCCO+64o1nbfffdp8LCQt14441KSkpqsm/kyJGO3n/jxo2KjY1t93WeeeYZnrx7BHB1Yqu1dpsxZoOkkcaYVGtt/iFOqR92wyK1AAAgqGbNmtWs7amnnlJhYaFmzpypfv36uXr/IUOcWZejT58+jlwHHVswFoPtEdjWtuHYEwPbrS7V4pqOu4ooAABwWv2486qqKt15553KzMxUVFSULr/8cklSYWGh/vGPf2jSpEnq1auXIiMj1a1bN5133nn65JNPWrxmS2PiZ82aJWOMFi1apJdfflknnHCCYmNjlZKSoksuuUQ7d+5stbbGFi1aJGOMZs2apTVr1ujss89WUlKSYmNjNWHCBC1btqzFmnbv3q0rrrhCaWlpiomJ0ciRI/X00083uV57ZWVlafr06UpLS1NUVJT69u2r66+/Xrt372527J49e3TLLbcoMzNTcXFxSkpKUmZmpi6//HJt3fptfLTW6umnn9a4cePUrVs3RUdHq3fv3po6dapeeOGFdtfcEbS7J94YM0TSAWtt7kHtPtVNYE2TtMxauz/QPlbSamtt1UHHT1Ldk18l6bn21hVsHfhZAAAAwCXTp0/XypUrdeaZZ+qCCy5QWlqapLqhMbfddpvGjx+vs88+W8nJycrJydHrr7+ud955R2+88YamTZvW5vs89NBDev3113XeeedpwoQJ+vTTT/XCCy9o7dq1WrNmjaKiotp0nc8++0z33HOPTjrpJF111VXKycnRK6+8otNPP11r1qxRZmZmw7F5eXkaN26csrOzNX78eI0bN065ubm6/vrrdcYZZxzeX1Qr3nzzTU2fPl3WWs2YMUN9+/ZVVlaWHn74Yc2bN09Lly5t+ASkrKxMJ598srZs2aIpU6bo3HPPlbVW27Zt07x58zRjxgwNGDBAknTbbbfp73//u/r376+LL75YiYmJ2r17t1auXKmXXnpJP/zhDx2pP5ScGE4zTdI/jDEfSdqiujXf01W34swASbmSrm50/N2ShgWWk6xfxmWEpEmB72+31rb8dhAAAKAD2bZtm9avX6/U1NQm7UOHDtWuXbuate/YsUMnnHCCbrrppsMK8fPnz9fKlSt1zDHHNLT913/9l55//nnNmzdPF198cZuu89Zbb+nJJ59s+MRAkh599FFdd911uv/++/XQQw81tN96663Kzs7Wb3/7W919990N7TNnztQJJ5zQ5tpbU1JSossvv1w1NTVatGiRTj311IZ9d999t37/+9/rmmuu0XvvvSdJ+uCDD7RlyxbNnDlT9957b5NrVVVVqbKyssmfqWfPnlq/fn2zeQb5+Yca3e0NToT49yX9r6STJR0rKUlSqeomtD4r6QFrbUGj45+V9ANJx0s6U1KEpD2qe7rrv6y1HztQEwAAcFC/378V6hLaLHv22UG711133dUsqEtSYmJii8f36tVLM2bM0IMPPqicnJw2j1+/4YYbmgR4Sbr66qv1/PPPa8WKFW0O8SeffHKTAC9JV155pX75y19qxYoVDW1VVVV6/vnnlZiYqD/+8Y9Njj/22GP105/+VI899lib7tmaefPmad++ffrRj37UJMBL0s0336xHHnlECxYsaPb3FBMT0+xakZGRioyMbNIWERGhsLCwZse29O/lRe0eE2+tXW+t/YW1dqS1NtVaG26tTbTWHm+tnXVQgJe19nFr7TnW2n7W2i7W2ihrbR9r7Q+9HOAto+IBADjifFeP9NKlS3XxxRerd+/eioqKalia8sEHH5SkFsezt2bMmDHN2nr37i1J2r9/f7uuExERofT09CbX+eqrr1ReXq4RI0YoPj6+2TmnnHJKm+/ZmlWrVkmSJk2a1GxfeHi4xo8fL0lavXq1JGnChAnq2bOnZs+erWnTpumBBx5QVlaWamubT7u89NJLlZ2drWHDhunWW2/V/PnzVVhY2O6aOxJXV6c5kjAmHgCAI09GRkaL7a+99ppmzJih6OhoTZkyRQMHDlRcXJx8Pp8WLVqkxYsXNxn+cSgHL28p1QVdSS2G2MO5Tv21Gl+nPvCmp6e3eHxr7Yej/h7du3dvcX99+4EDByRJCQkJ+uSTT3THHXfo9ddf17vvviuprmf9+uuv1x//+EdFRERIku69914NHDhQTzzxhGbPnq3Zs2crPDxcZ511lv75z39q0KBB7a4/1AjxDiHDAwA6s2AOUfGS1p6MevvttysyMlKfffaZhg4d2mTftddeq8WLFwejvO8tISFBUt1qMC1prf1w1A85ys3NbXF//eo0jYcm9erVS48//ristdqwYYMWLlyof//737rzzjvl9/t11113SZLCwsJ044036sYbb1ReXp6WLFmiOXPm6KWXXtIXX3yhL774os2TgTuqYCwxCQAAcETZvHmzjj766GYB3u/3a8mSJSGqqu2GDBmimJgYrVu3TsXFxc32O/FnOO644yTVLX95sJqamoZ7jBo1qtl+Y4yGDRumX/3qV1qwYIEkae7cuS3eJy0tTRdeeKFefPFFTZo0SVu2bNH69evbXX+oEeIBAAAc1q9fP23atEm7du1qaLPW6s9//rM2bNgQwsraJjIyUj/84Q9VWFiov/zlL032rV27Vs8880y773HBBRcoJSVFzz//fLO18++77z5t3bpVkydPbpjUun79emVnZze7Tv2nAvWr0FRWVuqDDz6QPWisc3V1tQoKCpoc62UMp3HIwT8oAADgyHXTTTfpuuuu03HHHafp06crIiJCS5cu1YYNG3TuuefqjTfeCHWJhzR79mwtXLhQ99xzjz799FONGzdOu3fv1osvvqizzjpLc+fOlc/3/fuDu3TpoieeeEIXXXSRJkyYoIsuukh9+vRRVlaW3nvvPWVkZOjRRx9tOP7999/Xr3/9a40bN05DhgxRWlqaduzYoXnz5snn8+k3v/mNJKm8vFyTJ09Wv379NHbsWPXt21cVFRVasGCBNm7cqPPOO6/ZJyReRIh3CBEeAADUu/baaxUVFaX77rtPTz/9tGJiYnTqqafqySef1CuvvOKJEJ+enq5ly5bpD3/4g95++219+umnyszM1EMPPaS4uDjNnTu3Yez893X++edr6dKl+tvf/qZ3331XhYWFysjI0HXXXafbb79dPXr0aDh26tSpmjlzpj766CPNmzdPRUVF6t69u6ZMmdIQ7iUpLi5Od999tz788EMtW7ZMc+fOVXx8vAYOHKiHH35YV155Zbtq7ihMZ+xBNsZkjRo1alRWVlbQ7vnhV3m64smVDa+ZAAQA8JKNGzdKUqfooYT7brvtNv3tb3/T/PnzNXXq1FCX44q2/k6MHj1aq1atWmWtHR2MuuoxJt4h/brGhboEAAAARzUe01/v888/1wMPPKCUlBRNmDAhBFVBYjiNYyLDv30/1D0xOoSVAAAAOGPMmDEaNGiQhg8frri4OG3atElvvfWW/H6/HnnkEUVHk3lChRDvkJZXiQUAAPCua6+9VnPnztXzzz+v4uJiJSUlaerUqbrllls0ceLEUJd3RCPEAwAAoEV33HGH7rjjjlCXgRYwJh4AAADwGEI8AAAA4DGEeBd0wlU7AQAAjhheWIKdEO8Qw8xWAICHmcD/yPx+f4grAUKvPsSbDhzwCPEAAEBRUVGSpNLS0hBXAoRe/e9B/e9FR0SIBwAAio+PlyTl5uaquLhYfr/fE0MKAKdYa+X3+1VcXKzc3FxJ3/5edEQsMQkAAJSSkqLS0lKVlZVpx44doS4HCLnY2FilpKSEuoxWEeIBAIB8Pp969+6tgoICFRcXq7Kykp54HHGMMYqKilJ8fLxSUlLk83XcQSuEeBdY8R89AID3+Hw+paamKjU1NdSlADiEjvv2wmOMOu7sZQAAAHQuhHgAAADAYwjxAAAAgMcQ4gEAAACPIcS7gMn8AAAAcBMh3iEd+Km8AAAA6GQI8QAAAIDHEOIBAAAAjyHEAwAAAB5DiAcAAAA8hhDvAhanAQAAgJsI8Q5hcRoAAAAECyEeAAAA8BhCPAAAAOAxhHgAAADAYwjxLrDMbAUAAICLCPFOYWYrAAAAgoQQDwAAAHgMIR4AAADwGEI8AAAA4DGEeFcwsxUAAADuIcQ7xDCzFQAAAEHiSIg3xtxtjPnAGLPdGFNujCkwxqw2xtxhjOnayjnjjDFvB44tM8asM8bMNMaEOVETAAAA0Fk51RN/k6Q4SQsk3S/p/0mqkTRL0jpjTO/GBxtjzpf0kaTxkl6T9G9JkZLulTTHoZoAAACATincoeskWGsrDm40xvxV0h8k3Srp+kBbgqT/k1QraaK19rNA++2SFkqaYYy5xFpLmAcAAABa4EhPfEsBPuDFwHZwo7YZkrpJmlMf4Btd44+Blz93oi4AAACgM3J7Yuu5ge26Rm2TAtv5LRz/kaQySeOMMVFuFuYmy+I0AAAAcJFTw2kkScaYWyR1kZQoaYykU1QX4Gc3OiwzsP364POttTXGmG8kDZM0QNLGQ9wvq5VdQw6v8vYzLE4DAACAIHE0xEu6RVJ6o9fzJV1urd3bqC0xsC1s5Rr17UkO1wYAAAB0Co6GeGtthiQZY9IljVNdD/xqY8w51tpVbbxMfZ/2IQelWGtHt3iBuh76UW28HwAAAOAproyJt9busda+JukMSV0lPdNod31Pe2KzE+skHHQcAAAAgEZcndhqrd0maYOkYcaY1EDzV4HtUQcfb4wJl9RfdWvMb3WzNjcxrxUAAABucnt1GknqEdjWBrYLA9tpLRw7XlKspGXW2kq3C3MS81oBAAAQLO0O8caYIcaYjBbafYGHPaWpLpTvD+x6WVK+pEuMMWMaHR8t6S+Blw+3ty4AAACgs3JiYus0Sf8wxnwkaYukfapboWaC6paJzJV0df3B1toiY8zVqgvzi4wxcyQVSDpPdctPvizpBQfqAgAAADolJ0L8+5L+V9LJko5V3dKQpapbB/5ZSQ9Yawsan2CtnWuMmSDpNknTJUVL2izp14HjGVYOAAAAtKLdId5au17SL77HeUslndXe+wMAAABHmmBMbD3i8EECAAAA3ESId4gxrE8DAACA4CDEAwAAAB5DiAcAAAA8hhAPAAAAeAwh3gVMawUAAICbCPEOYVorAAAAgoUQDwAAAHgMIR4AAADwGEI8AAAA4DGEeBfwwFYAAAC4iRDvEB7YCgAAgGAhxAMAAAAeQ4gHAAAAPIYQDwAAAHgMIR4AAADwGEK8CyzL0wAAAMBFhHiHGLE8DQAAAIKDEA8AAAB4DCEeAAAA8BhCPAAAAOAxhHgXMK0VAAAAbiLEO4V5rQAAAAgSQjwAAADgMYR4AAAAwGMI8QAAAIDHEOLdwMxWAAAAuIgQ7xDDxFYAAAAECSEeAAAA8BhCPAAAAOAxhHgAAADAYwjxAAAAgMcQ4l3A4jQAAABwEyHeISxOAwAAgGAhxAMAAAAeQ4gHAAAAPIYQDwAAAHgMId4F1jK1FQAAAO4hxDvEGKa2AgAAIDgI8QAAAIDHEOIBAAAAjyHEAwAAAB5DiAcAAAA8pt0h3hjT1RhzlTHmNWPMZmNMuTGm0BizxBjzM2OM76Dj+xlj7Hd8zWlvTaHG2jQAAABwU7gD17hI0sOSdkv6UFKOpHRJF0p6TNKZxpiLbPN1F9dKmtvC9dY7UFPQsTYNAAAAgsWJEP+1pPMkvWWt9dc3GmP+IGmFpOmqC/SvHHTeGmvtLAfuDwAAABxR2j2cxlq70Fr7RuMAH2jPlfRI4OXE9t4HAAAAQB0neuK/S3VgW9PCvh7GmGsldZW0T9Jya+06l+sBAAAAPM+1EG+MCZf008DL+S0cMiXw1ficRZIus9bmtPEeWa3sGtLGMl3RbPQ/AAAA4CA3l5icLWm4pLette82ai+TdJek0ZKSA18TVDcpdqKkD4wxcS7W5QrDzFYAAAAEiSs98caYGyTdLOlLST9pvM9amyfpTwed8pEx5gxJSySNlXSVpPsPdR9r7ehW7p8ladThVw4AAAB0fI73xBtjfqG6AL5B0mnW2oK2nGetrVHdkpSSNN7pugAAAIDOwtEQb4yZKelfqlvr/bTACjWHY29g67nhNAAAAECwOBbijTG/k3SvpDWqC/B53+MyJwa2W52qKxQsz2wFAACAixwJ8caY21U3kTVL0unW2vzvOHasMSayhfZJkm4KvHzOibqCyfDMVgAAAARJuye2GmMuk3SnpFpJH0u6wTRfqiXbWvtU4Pu7JQ0LLCe5I9A2QtKkwPe3W2uXtbcuAAAAoLNyYnWa/oFtmKSZrRyzWNJTge+flfQDScdLOlNShKQ9kl6U9C9r7ccO1AQAAAB0Wu0O8dbaWZJmHcbxj0t6vL33BQAAAI5Ubj7sCQAAAIALCPEusCxOAwAAABcR4h3SfC4vAAAA4A5CPAAAAOAxhHgAAADAYwjxAAAAgMcQ4l3AvFYAAAC4iRAPAAAAeAwhHgAAAPAYQjwAAADgMYR4AAAAwGMI8W5gZisAAABcRIh3CE9sBQAAQLAQ4gEAAACPIcQDAAAAHkOIBwAAADyGEA8AAAB4DCHeBZblaQAAAOAiQrxDjFieBgAAAMFBiAcAAAA8hhAPAAAAeAwhHgAAAPAYQrwLLPNaAQAA4CJCvEMM81oBAAAQJIR4AAAAwGMI8QAAAIDHEOIBAAAAjyHEAwAAAB5DiHcBi9MAAADATYR4h7A4DQAAAIKFEA8AAAB4DCEeAAAA8BhCPAAAAOAxhHgXWMvUVgAAALiHEO8QY5jaCgAAgOAgxAMAAAAeQ4gHAAAAPIYQDwAAAHgMId4FTGsFAACAmwjxDmFaKwAAAIKFEA8AAAB4DCEeAAAA8BhCPAAAAOAx7Q7xxpiuxpirjDGvGWM2G2PKjTGFxpglxpifGWNavIcxZpwx5m1jTIExpswYs84YM9MYE9bemgAAAIDOLNyBa1wk6WFJuyV9KClHUrqkCyU9JulMY8xF1tqGRVuMMedLekVShaQXJBVIOlfSvZJODlzTsyzL0wAAAMBFToT4ryWdJ+kta62/vtEY8wdJKyRNV12gfyXQniDp/yTVSpporf0s0H67pIWSZhhjLrHWznGgtqAxLE8DAACAIGn3cBpr7UJr7RuNA3ygPVfSI4GXExvtmiGpm6Q59QE+cHyFpD8GXv68vXUBAAAAnZXbE1urA9uaRm2TAtv5LRz/kaQySeOMMVFuFgYAAAB4lRPDaVpkjAmX9NPAy8aBPTOw/frgc6y1NcaYbyQNkzRA0sZD3COrlV1DDq9aAAAAwDvc7ImfLWm4pLette82ak8MbAtbOa++PcmtwgAAAAAvc6Un3hhzg6SbJX0p6SeHe3pge8g1Xqy1o1u5f5akUYd533YxzGwFAABAkDjeE2+M+YWk+yVtkHSatbbgoEPqe9oT1bKEg44DAAAA0IijId4YM1PSvyStV12Az23hsK8C26NaOD9cUn/VTYTd6mRtAAAAQGfhWIg3xvxOdQ9rWqO6AJ/XyqELA9tpLewbLylW0jJrbaVTtQEAAACdiSMhPvCgptmSsiSdbq3N/47DX5aUL+kSY8yYRteIlvSXwMuHnagrlCyPbQUAAIBL2j2x1RhzmaQ7VfcE1o8l3dDCJM9sa+1TkmStLTLGXK26ML/IGDNHUoHqnvqaGWh/ob11AQAAAJ2VE6vT9A9swyTNbOWYxZKeqn9hrZ1rjJkg6TZJ0yVFS9os6deSHrB0YwMAAACtaneIt9bOkjTre5y3VNJZ7b0/AAAAcKRx82FPAAAAAFxAiAcAAAA8hhDvEkb1AwAAwC2EeAc1X5QHAAAAcB4hHgAAAPAYQjwAAADgMYR4AAAAwGMI8S5hXisAAADcQoh3EPNaAQAAEAyEeAAAAMBjCPEAAACAxxDiAQAAAI8hxAMAAAAeQ4h3ibWsTwMAAAB3EOIdZAzr0wAAAMB9hHgAAADAYwjxAAAAgMcQ4gEAAACPIcS7hGmtAAAAcAsh3kFMawUAAEAwEOIBAAAAjyHEAwAAAB5DiAcAAAA8hhDvEh7YCgAAALcQ4h3EA1sBAAAQDIR4AAAAwGMI8QAAAIDHEOIBAAAAjyHEAwAAAB5DiHeJFcvTAAAAwB2EeAcZsTwNAAAA3EeIBwAAADyGEA8AAAB4DCEeAAAA8BhCvEss81oBAADgEkK8k5jXCgAAgCAgxAMAAAAeQ4gHAAAAPIYQDwAAAHgMIR4AAADwGEK8g5jXCgAAgGAgxAMAAAAeQ4gHAAAAPMaREG+MmWGMedAY87ExpsgYY40xz7VybL/A/ta+5jhREwAAANBZhTt0nT9KOlZSiaQdkoa04Zy1kua20L7eoZoAAACATsmpEH+T6sL7ZkkTJH3YhnPWWGtnOXT/DsfaUFcAAACAzsqREG+tbQjtxhy5a7QcwX90AAAABJFTPfHfRw9jzLWSukraJ2m5tXZdCOsBAAAAPCGUIX5K4KuBMWaRpMustTltuYAxJquVXW0Zkw8AAAB4UiiWmCyTdJek0ZKSA1/14+gnSvrAGBMXgroAAAAATwh6T7y1Nk/Snw5q/sgYc4akJZLGSrpK0v1tuNboltoDPfSj2llqu1gxsxUAAADu6DAPe7LW1kh6LPByfChr+b6MmNkKAAAA93WYEB+wN7BlOA0AAADQio4W4k8MbLeGtAoAAACgAwt6iDfGjDXGRLbQPkl1D42SpOeCWxUAAADgHY5MbDXGXCDpgsDLjMD2JGPMU4Hv8621twS+v1vSsMBykjsCbSMkTQp8f7u1dpkTdQEAAACdkVOr04yUdNlBbQMCX5K0TVJ9iH9W0g8kHS/pTEkRkvZIelHSv6y1HztUU0hZFqcBAACASxwJ8dbaWZJmtfHYxyU97sR9OxrD4jQAAAAIgo42sRUAAADAIRDiAQAAAI8hxAMAAAAeQ4h3CfNaAQAA4BZCvIOY1woAAIBgIMQDAAAAHkOIBwAAADyGEA8AAAB4DCHeJZZHtgIAAMAlhHgHGR7ZCgAAgCAgxAMAAAAeQ4gHAAAAPIYQDwAAAHgMIR4AAADwGEK8S1ibBgAAAG4hxDuItWkAAAAQDIR4AAAAwGMI8QAAAIDHEOIBAAAAjyHEu8QysxUAAAAuIcQ7iZmtAAAACAJCPAAAAOAxhHgAAADAYwjxAAAAgMcQ4t3CxFYAAAC4hBDvIOa1AgAAIBgI8QAAAIDHEOIBAAAAjyHEAwAAAB5DiAcAAAA8hhDvIGO+ndrqtyxPAwAAAHcQ4h3UKMOzwiQAAABcQ4h3kK9Rirf0xAMAAMAlhHgH+Rr1xPvJ8AAAAHAJId5R9MQDAADAfYR4B/kYEw8AAIAgIMQ7yDQZTkOMBwAAgDsI8Q4yTYbThLAQAAAAdGqEeAc17okHAAAA3EKIdwkd8QAAAHALId5BjTviWZ0GAAAAbiHEO8gwngYAAABBQIh3CR3xAAAAcIsjId4YM8MY86Ax5mNjTJExxhpjnjvEOeOMMW8bYwqMMWXGmHXGmJnGmDAnagIAAAA6q3CHrvNHScdKKpG0Q9KQ7zrYGHO+pFckVUh6QVKBpHMl3SvpZEkXOVRXUDUeTUNPPAAAANzi1HCamyQdJSlB0s+/60BjTIKk/5NUK2mitfZn1trfSBopabmkGcaYS6CqogYAACAASURBVByqK6gYEg8AAIBgcCTEW2s/tNZusm1bkmWGpG6S5lhrP2t0jQrV9ehLh3gj4AWWRSYBAADgklBMbJ0U2M5vYd9HksokjTPGRAWvJGfwxFYAAAAEg1Nj4g9HZmD79cE7rLU1xphvJA2TNEDSxu+6kDEmq5Vd3zkm3y0MpwEAAEAwhKInPjGwLWxlf317UhBqcQ0d8QAAAHBLKHriD6W+P/uQOdhaO7rFC9T10I9ysqi24ImtAAAACIZQ9MTX97QntrI/4aDjPIMntgIAACAYQhHivwpsjzp4hzEmXFJ/STWStgazKKfRDw8AAAC3hCLELwxsp7Wwb7ykWEnLrLWVwSvJGU2H04SsDAAAAHRyoQjxL0vKl3SJMWZMfaMxJlrSXwIvHw5BXe3mb5TcK2tqQ1gJAAAAOjNHJrYaYy6QdEHgZUZge5Ix5qnA9/nW2lskyVpbZIy5WnVhfpExZo6kAknnqW75yZclveBEXcGWva+s4ftnl2/T7OkjQlgNAAAAOiuneuJHSros8DU10DagUduMxgdba+dKmqC6hztNl/QrSdWSfi3pkjY++bVDm7Nye6hLAAAAQCflSE+8tXaWpFmHec5SSWc5cX8AAADgSBKKMfEAAAAA2oEQDwAAAHgMIR4AAADwGEI8AAAA4DGEeAAAAMBjCPEAAACAxxDiAQAAAI8hxAMAAAAeQ4gHAAAAPIYQDwAAAHgMIR4AAADwGEI8AAAA4DGEeAAAAMBjCPEAAACAxxDiAQAAAI8hxAMAAAAeQ4gHAAAAPIYQDwAAAHgMIR4AAADwGEI8AAAA4DGEeAAAAMBjCPEAAACAxxDiAQAAAI8hxAMAAAAeQ4gHAAAAPIYQDwAAAHgMIR4AAADwGEI8AAAA4DGEeAAAAMBjCPEAAACAxxDiAQAAAI8hxAMAAAAeQ4gHAAAAPIYQDwAAAHgMIR4AAADwGEI8AAAA4DGEeAede2yPhu8HpMaFsBIAAAB0ZoR4B11z6oCG7/cUVYSwEgAAAHRmhHgHGfPt96VVtbLWhq4YAAAAdFqEeAc1DvGSRIYHAACAGwjxDjo4tJPhAQAA4AZCvIv8dMUDAADABYR4Bx2c2QnxAAAAcEPIQrwxJtsYY1v5yg1VXe1hDxpAQ4YHAACAG8JDfP9CSfe10F4S7EKc0GxMPCEeAAAALgh1iD9grZ0V4hocc3BmZzgNAAAA3MCYeAcdvC48IR4AAABuCHVPfJQx5seS+kgqlbRO0kfW2tq2nGyMyWpl1xCH6jsszXviQ1EFAAAAOrtQh/gMSc8e1PaNMeYKa+3iUBTUHs063gnxAAAAcEEoQ/yTkj6W9IWkYkkDJP1S0jWS3jHGnGStXftdF7DWjm6pPdBDP8rZctuC4TQAAABwX8hCvLX2zwc1rZd0nTGmRNLNkmZJ+kGw62oP1okHAABAMHTEia2PBLbjQ1rF99AzOabJa8bEAwAAwA0dMcTnBbZxIa3ie+ieeHCIJ8UDAADAeR0xxJ8U2G4NaRUOeH3NrlCXAAAAgE4oJCHeGDPMGJPSQntfSf8KvHwuuFU577XVO0NdAgAAADqhUE1svUjS740xH0r6RnWr0wyUdLakaElvS/rvENXmmA27i0JdAgAAADqhUIX4DyVlSjpOdcNn4iQdkLREdevGP2sPfvwpAAAAAEkhCvGBBzl57mFOAAAAQEfQESe2AgAAAPgOhHgAAADAYwjxAAAAgMcQ4gEAAACPIcQDAAAAHkOIBwAAADyGEO8ylrsHAACA0wjxLvOT4QEAAOAwQrzLtu0rDXUJAAAA6GQI8S677bX1oS4BAAAAnQwh3mXLt+4LdQkAAADoZAjxAAAAgMcQ4h0WFxkW6hIAAADQyRHiHXb9aYOatZVV1YSgEgAAAHRWhHiHnXF0erO2d7/IDUElAAAA6KwI8Q6LDG/+V7ppT0kIKgEAAEBnRYh3WEZidLO2hxZtCUElAAAA6KwI8Q6LCm95Yivj4gEAAOAUQrwLJmZ2a9Z266ufh6ASAAAAdEaEeBc8cdnxzdrmrdmlpZvzQ1ANAAAAOhtCvAt8PtNi+6WPfRrkSgAAANAZEeKDbNkWeuMBAADQPoT4IHt11c5QlwAAAACPI8QH2ctZO7S/tCrUZQAAAMDDwkNdwJHouLsWSJKuPLm/bj9nqIxpeQw9AAAA0BJ64kPoiaXfqP+tb+vsBz5Wdn5pqMsBAACARxDiXXLhqJ5tPvaLXUWa+N+L9OjiLdpfWqW9xZX6fEdhk2P2lVQ6XSIAAAA8iuE0Lvmfi0cqzBi9lLWjzef8/Z0v9fd3vmx4PW5gV43snaRPtu7TqpwDDe1nj+iuP51ztJ5Y8o0GpXXRRWN6f+d1K2tq9cba3aqsqdUPx/RWeJh33rvV1PqVW1ShXsmxoS4FAACgwzDW2lDX4DhjTNaoUaNGZWVlhboUjbzzPR0oq3b9Pu/OHK+3P9+t+z/YJEmaP/NUPbJoi2r8VnlFlVqRXdBw7Ka/nqmIMJ8Ky6r1wMJNSoqJ0C9OG6SSqhot3JinUwenqmuXqIbjdx0oV1JshGIjW3/PV/9zdDjj+8uravXO+t0a1iNRmRnxzfbX+q3OvP8jfb2nRLeeOUTXThjY5mt/H0UV1Xrn890a0y9FA7t1ceSa1lrmPAAA0ImNHj1aq1atWmWtHR3M+9IT77KnrzhB5/97qev3mXrfR01eT7vv41aPHXzbO83a9hRX6LlPchpe3zT5KJVV1ejRj7Y2tK294wz5/VYbdhdpbP8UffpNQZMHWGWmx+v5a05USlyksvNL9fnOQp05PKOh5z+/pFLb9pVp+ZZ8XTdhoKbcu1g79pdLkn52Sn8VV1Tr5jMylZ4QLUl6f+Mefb2nRFLdpxTXThioHfvL9MjiLTq2V5IuGtNbeUUVio0K1+qc/cpIiNbg9OZvBupV1fgVEWaahOqiimolREdIku6Y94VeW71TybERWn7r6YqOCGv1Wo3V1Pq1raBMd725QQNSu+j2c4aqssavnz6xQrsLy/XwpaM1vGdim67V2UN/ZU2twn0+hbXyQDQAANA29MQHwfqdhTrnwSWhLiNofjM1U/9496t2XeNXkwbpwYWbm7Qt/s1ETfjHoobXV5zcT08uzW5yzMKbJ6h7Yoxyiyp02RMr5LdWC26aoHe/yNXt89arT0qsXvn5OO0pqmhyrZa8/suTNaJXkiRpzfYDeveLXJ2WmaZ+XWOVlhCtb/JL9esX1+iLXUWqqvE3nHd09wSlJURp0Vd7G9qyZ5+tmlp/k6FMRRXVig4PU2S4T7V+q7Mf+Fhf5hbr6StP0JJNe1VSWaPfTRuipNhI+f221ScB1+/7dOs+vbFul0b1SdYFI3uqpKpG2wvKNLBblyZvSCqqa7Vjf5kGpbX+hudg1lrtPFCuZ5Zv0wn9UjT56PQ2n1tv7fYDuvzJFeoSHa55vzhFKXGRh30NAAA6mlD1xBPig+TZ5dm6fd4XoS4DHUx8dLjS4qO0ZW/bVieKjwpXcWWNJOml605SUXm1fvb0Z82OO6Znoj7f+e3k6F+eNkhXntJfZVU1OufBJTpQVq3ULlGaMbqXsvNL1bVLpF5dtVOP/GS0xg9Obfg0oKbWr3vf/1r//nBLk+sv/f0k9UiM1tLN+xQTGabRfZMb9lVU1yq/pFKpXaIU7jOa8chyrdl+oMn500f10j8vPrbhdWVNrWpqreKi6j4crPVbVdX4FRMZppx9ZXrxs+2amNlNPZNjlJEQ/Z2fVjT+NMNaq8oaf5M3MW+t263XVu/Q4PR4XXZSP2UkRn/3X7qk5Vv2ae2OA7p4TO9mbz78fitj2j6UrKK6Vj5jFBnuzNyUyppaGTl3ve+r/v8lVbV+ffx1vkb0TlRa/KH/bgHA6wjxDuqIIV6Spt33kb7MLQ51GYAj7p5+jH73yucNr4f3TNCYvil6all2m84/tneSLh3bR8UVNbrrzQ0NbbeccZR+8viKhuPCfUY1/m//O3VMz0T99QfD5bdS/9Q4bc4r1vsb8/TG2l0Nw7N+cdpA/WrSYJ39wMdN3iCdPiRNH3yZ16SOHxzXUzefcZQ+2JinyUena+32A/r3h5tVVePXnGtO1Nb8Ul30yHJJ0oBucZoyNF29UmK1eU+x0hOjdc/8uk+d/nTO0erfLU7vb9ijc4/toVF9khURVhfs6wP+h1/l6YonV0qS/nPVWD2/crsGpMZp5uTBqqzxa3NeiYb1SJC1kt/aJp/c+P1Wu4sq9P8+2aaFX+apb9dY/WrS4IZP+Y5K76J3bhyvMJ9pePPy5NJsjemXrOTYSFXX+jW0e8J3/pv4/Va11mrr3lIdld6loe6C0iolx0Y0eXO04psC9UuNU3pCtDbnleiqp1fK5zMakhGvtz/PVbf4KC393STH31zU+m2bh2OVVdVo1bYDOr5/sqLCDz08bueBctXWWvXp+u1E+o4yxK2sqkYxEWEhr2V7QZm6J0arsLxaK74p0ITMbt85Xwo4EhDiHdRRQ3xhebXO/9cSZe8rC3UpAIKoa1ykThrYVW+u2+36vXomxWjngfLvPGby0DRNH9VL4WE+lVfXKikmQj99YkWz4+Zcc6KWbs5vGNr26vXj9I/5X2n51n1tquXkQV2VnV+m2885WicOSNGTS7PVPTFal5zQR1v3lmjSPxdLkp664niN6JWkj77eq9My07Qlv0TH9EzUgbJqLduSr9zCCn2ZW6zXVu9suHb/1DjdPX2EhnaP1wMfbNI3+aW649xhiosK1/aCsmZzkR758Sh1T4zRV7nFiorw6X8WfK1zR/TQxMxuGtUnWet2FuqCwDnPXHmC+naN1f0fbNLCL/N0+bh+WrZ5n0oqa3TVqf1lrbRjf7lO6J+i4T0TFB+YV1Mva1uBpj+8XD87pb+uPnWA3vp8tyZmdlOv5BhFhvlkreTzGe0vrVKX6HBV1fiVV1yp5Vv26Yxh6Zq7eqfue3+TSiprNOXodE0dlqFbXlorSXrzV6c0mWMzf32urnsuS3/7wTEqqazW4PR4nTIoVfklleqeGNPs38Tvt/pk6z7tL6tWeXWtzhnRXdERYar1W23KK1Zmerwqa/xa/PVefZ1brLlrdur8kT11w+mD9cjiLZr9zpc6Kr2Lqmr8yt5XpqnD0vXoT8a0+jOQs69M//vxFp0yqJumDc9o089NaWWNrKTYiLAmb2ZbGlb4VW6xnvtkm0oqa3TT5KOavAGTmr/pW7+zUH+at15H90jQXecPb/LGtPEnkOFhPlXV+OW3ttX5UTW1fm3YXaRhPRLb/MayqKJa+cWVGtCtizbnlai8qlY9k2OUHBuhnIIyvb5mlzIz4jXl6PQmb9iWbs7Xsi35unRsX/VIav7v+n1VVNdq7fYDWrvjgC4a3VvJcZF68INN+s+KHN14+mCFh/mUs69UV57SX0mxrQ+BtNaqqLxGibERrR7TXpvzihUZFtbs37gjIMQ7qKOG+MaKKqo1YtZ7kqQ+KbHKKSDYA4CXXTq2j77MLVbWtv2u3yszPV5f7Tn0J7v1n3j1To7V6u37Gz45asxnJL8DUeDsY7pr3KCuWpNzQKP6JqukokZ/fXtjw/7TMrvpw8BcoRtPH6zSyholxUbov9/7+juv+/J1JymnoEy/fnFtm+pIiA5XekK0NuWVNLQd2ztJpw9J0/8saHqvyUPTtPDLvGZ//r5dY7Ut0OE2tn+KeqfEasveEk0emq7Lx/WT31odE/h/uCSdP7KHPt9RqCnD0hXh8ym1S6SmDe+uoopqvbs+V6+s2qGcgrLD+nv+/ZlDNCA1TjsPlOvPb2xoaO+VHKNzRvTQI4vrhjm+eO1JOiq9S0PIrqiu1ZR7F2t7QbmG9UhQ/9Q4XTCyp7onRevxJd+opKJGN5w+WP9876uGf4964wZ21bItLb9Jf/yyMVqwYY8uHdtXx/RKbDLPq9/v32o4bv2fp6q6xq/YqDAVlFbJZ4xW5+xXRbVfW/NL9UBgFb16qV0idd2Egbrq1AHanFeihxdt0SUn9FZsZJjWbi/U08uym/2sXzN+gG45I1MRYUbvfrFHb6zbpdMy03TesT1CMrSQEO8gL4R4qe6d6/6yaqXERcpaq/P+tVSf7yzURaN7Hdb68gAAAJA+/u1p6p0S3N76UIV47zz1pxMyxjRMkjPG6JWfj9P8mafqnhkjlD37bP3lguHymbrVTm6afFSz83smxWj2hccEu2wAAIAO6dR7Pgx1CUHDbJQOJDLcpyEZ3048+/GJfTVteIZSYiPl8xm9vnantuwt1cBucfrLBcc0jMW85IQ+kup69r/YVaS+XWPVJSq8YTzdxt1FSu0SpeTYCG3ZW6roCJ9yCsrkM0Z9u8bqn+99rbT4KJ0/sqd6Jsfoo6/36theSRr/j6a/CA9fOkpnHtNdheXVunfB1w0TGB++dJQmH52uv7/9pV5YmaOKGr8e/NFxSomLVM6+Mv32lXWS6j5qHpzWRbMafSx4z/QRDfvb6tGfjFZ1rV+//M9qSVJ0hE8V1f5DnAUAANB5MJzGQ/KKK7Rgwx5NzExTTwcntrSmrKpGuw5UqFt8lKpr/Upt9BTXw1U/dq7Wb/XO+t2KCPNpytD0hklKecUV6tYlSn6rhglCRRXVemZZtnqnxOq8Y3vom/xS9UqObTLerbrWr4jAmLzyqlrtLa5smPSyYVeR3tuQq0cXb1V5da0k6ccn9tHt5xytyDCflm7ep3U7D2jG6F7Kyt6vYT0SlRgToWPvfE+tefBHx6myxq9TB6fq/32yTf9ZsV35JZU6f2QPXXBcT1XV+PWHVz/XvtIqSXWrpFw7YaASoiM0d/VOvblut97fuEdHd0/QjZMH69pnv/0ZTe0SpfySyu/9dwwAAOqezRJMjIl3UGcN8fh+tu4t0d/f+VKD07roN1MzD7lE284D5Xph5XadMihVJ/RPOez7fbGrULe++rkGpMbpfy4e2epDmiRp275S/e3tjaqutfrDWUPUIylGI+9cIEm6YdIgde0SpWnDMpQcF6nCsmot35qvsf27qtZaxUWGa1dhue56c4N6JcfoT+cMU63fKircp6pav1ZmF2h03+Qmy79tzivR0s35umhML8VGhitnX5l++sSnyt5Xpm7xUbpodC/dNOWohjdGkrSvpFJrdxzQ4LR4lVbVKCE6QtsLyrSnuFKpcZEa1jNR0RE+ffx1vrYVlOm0zG6qrPHrsY+/kZVVfFS4LhvXT12iwrV9f5liI8N122ufa1XOgWZ/H41dM36ALh3bRwWlVVq7/YDW7SjUq4HVSTISotUrOUY1fttsDfp6144f0PDE4bOOydDa7YWHXLWlLRJjInTT5MFNPlECAHQMt59ztH52Sv+g3pMQ7yBCPLyuqsYf8of3BEvjP+tXucX6bFuBzhnRQ4kxbVuqbNu+UiXFROp/Fnylp5dv0xlHp+vhH49uWC+9fjm/ermFFdqYW6R12wv1xrpdumnyURrZJ0np8VH6ak+xdu4vV7/UOO06UK7xg7s1nFtQWqWkmIiG1xt2FWnOyhydOby7ThrYVVLd0ng79pcrPjpcqV2iFBFmtHTzPj3wwSatyC5QapcozblmrLrGRenKp1fKZ4zumTFCcZHh8vnqHioVHx2uxV/t1TG9kjQxs5s255XojbW79PqaXZo+upcyM+KVV1Spq8f314srt+uTrQW64fTBCg8z+ia/VFHhPt3y0jolxoTrP1efqPSEaL2ctaNhicLBaV1U47f63bRMTRveXeVVtYoK92ntjgPKzIhXjd8qwufTiuwCbckr0b7SSg3vkaixA7qqS1S4wnxGr67aoTvf3KCeSTGaMbqX/NYqITpCFxzXU6tzDshaq6HdE3TXmxv0zb5S5RVVKr+kUkMy4pVTUKb9ZdVN/g3/csFwpSdEKyrcp1dX7dDcNbuUmR6vHfvLVFpV23BcS8tndouP0t7iuk+wfnRCb4X5jJ77JKfNP39S05VI6l03YaB+dkp/Hf/X95u0h/mMatu4xMjUYel694s9kurmNuUUlKkk8LA2J506OFU/nzBQ//XYp45fG/CaYPfCS4R4RxHigSNT4+FVaKqiurbV9a6/j5bW7G7reS98tl3FFdWamJmmiupajeiVdMjz6t/szV+fq+Vb8nXusT00pl/dJ2UtPZCpqsYvK9vsIU/vb9ijx5Zs1e/PHKohGfEyRk2OOfhara1/XVxRrbjIcBVX1CghJlx+K/3fx1tVXFGt6yYMVHx0hKy1Wr39gHonx6pbfN1wxMqaWkWG+WSMUVFFteIDTymu8Vu9sXaXPt6Ur0vH9lFSbIR6JccqOqJumb6qGr8yEqO1v7RKpVU1DcMbC0qrGtYNX7v9gJ5fkaPkuEj1SYnVmL7JGpTWpcW10KW6N51LN+crJjJMeUWVOjuwZny9F1bm6I21u3XdhIEa0y9Za7cfUFJspDIz4ptca9mWfK3fWajj+iQrKtzXsOZ//bMHJg9N08C0Lvrd1CEN677vLixXfHSEYiPCGn6O1u8s1JLN+Tp5YKrO/VfdQ8xmX3iMLjmhj4oqqvX+hj0a0StJg9Lq1qnfV1qpN9bukrXSmH7JOio9Xht3F2tvcaV+8Z9VkqQLR/VUSUWNJh+drovH9FZ1rV+rcw5ow65CTR2eoajwMCXFRGjOyu0qqqjW+MHd1D0xWjGRYXrow83akl+qCJ9RQkyEzj6mu+at3aWXs3ZIVvr1GUfpwlE9lZW9Xw8v3qIfj+2rC0f1VK21mr8+V6cO7qYwn9EnW/fpxAFdtWRTvkora/TIR1sUGebTuIGpSkuI0pnDMxQZ7tODCzdrQGqcLhvXT/PX56pncozyiip015sbdVyfJP31B8coITpcN7+4Vq+u3qnULpHyGaNJQ9L022lDFOYzWvFNgT7etFfnj+ypsqoaZW3br55JMfr0mwLll1Sqf2qcfjt1iGIiw2StVWF5tcbf86GKKmp0fL9kXTiql97+fLcmDUnTFSf3V2F5tcJ9RrPf+VLPfrKt4WcjMsynhbdM0MWPLNeuwoqG9vNH9tAvTxukBRv3qHtitJ77JEd/Pm+YYiPDtH1/ubpEhemJpdmKDPMpMyNes9/5UpI0aUiabjt7qN5et1urcvZr1nnDlNolSre++rkWf71XheV1b/yH9UhQYkxEi8tgRob59PLPT2rTf0+cdkSGeGNML0l3Spomqauk3ZLmSvrz/2/v/oPlKus7jr+/Cb9iSCJgAEdsCZhALP3DagGh5UeojFo6xRY7tiMFWmwZbYHKTFFba3BUELGilo5aFMKP1gKt0h8Uo2CgkDpqrW3VQCJJNNZAMBHILxKSPP3jeTbZbPbsvXfvzd09u+/XzM7JPc9z7j77yd17vvfZ8yOl1PWFdi3iJUnqre07dnHg1OjqLrM7dyV27krj+kRyLHf3HWZbt+/kuedf4KiZh3Tst2PnLhLsM1GyZsMWjjlsGtt27BrzRMHKpzfxkhkHM/OQsd8kateuxJLl6zjx6JkTegOsbvSqiO/Z1Wki4nhgKXAkcC/wGHAycAXw+og4PaU0utsCSpKkvjKeAnzqlBh3AW4BPzrTDprKtINGLr4PqPiUs3FN9m4+6Ttu9qFj3qZhypRgwYlHdb39IOjl585/TS7gL08pnZ9SeldKaQHwMeAE4IM9HJskSZLUt3pSxEfEccC5wGrgppbm9wGbgQsjYvokD02SJEnqe72aiV9QlotTSnvdpSeltBF4FHgRcOpkD0ySJEnqd706Jv6Eslxe0b6CPFM/D3ig6ptERNWZqyd2PzRJkiSpv/VqJn5WWT5b0d5YP/nXCZIkSZL6XM+uTjOCxinlHa9/WXUpnzJD/wsTPShJkiSpH/RqJr4x0z6ron1mSz9JkiRJRa+K+MfLcl5F+9yyrDpmXpIkSRpavSriv1qW50bEXmOIiBnA6cBW4GuTPTBJkiSp3/WkiE8pPQEsBo4F3tHSfA0wHbgtpbR5kocmSZIk9b1entj6dmAp8ImIOAdYBpwCnE0+jObPejg2SZIkqW/16nCaxmz8a4BbycX7VcDxwCeA16aU1vdqbJIkSVI/6+klJlNKa4BLejkGSZIkqW56NhMvSZIkqTsW8ZIkSVLNWMRLkiRJNRMppV6PYcJFxPpp06YdPn/+/F4PRZIkSQNs2bJlbN26dUNK6YjJfN5BLeJXATOB1ZP81CeW5WOT/Lx1Z27dMbfumFt3zK075tYdc+uOuXVnvLkdCzyXUpozMcMZnYEs4nslIv4TIKX06l6PpU7MrTvm1h1z6465dcfcumNu3TG37tQ1N4+JlyRJkmrGIl6SJEmqGYt4SZIkqWYs4iVJkqSasYiXJEmSasar00iSJEk140y8JEmSVDMW8ZIkSVLNWMRLkiRJNWMRL0mSJNWMRbwkSZJUMxbxkiRJUs1YxEuSJEk1YxE/ASLimIj4XET8OCK2RcTqiLgxIg7r9dgmSkQcERGXRsQXIuL7EbE1Ip6NiEci4vcjou3PUkScFhH3RcSGiNgSEf8TEVdGxNQOz3VRRHw9IjaV51gSEed16D8tIq6JiMcj4vmIWBcRd0XE/Il47ftDRFwYEak8Lq3oY3ZARPxyRPxDRKwt76+1EbE4It7Ypq+ZARHxqyWjH5X36sqIuDsiXlvRfyhyi4gLIuKTEfHvEfFcef/dMcI2fZlNTOJ+Zyy5RcTciLg6Ih6MiDURsT0inoqIeyPi7BGeZ2hzq9j+s7FnP/GKDv2GPrfILiqvfUPk33uryuuaV7FN/XNLKfkYxwM4HngKSMAXgeuAB8vXjwFH9HqME/Q6Lyuv6cfAncC1wOeAZ8r6eyg3D2va5teBHcAm4LPAR0omCbi74nluKO1rgI8BNwHry7o/atP/YOCR0v4N4MPA3wIvAJuBU3qdXZsxv7zktrGM+9I2fcwuj/HPy/ieBm4BPgR8poz3ejNr+5o+XMb3E+Dm8jvpY04KDgAACN9JREFUHmA7sAt467DmBny7jGEjsKz8+44O/fsyGyZ5vzOW3IDPl/bvAp8m7yv+seSYgMvNbVTb/lrTtgl4hblV9j8E+Oem8fxV+blbBKwEzhvU3CYs9GF9AF8q/yF/3LL+L8v6T/V6jBP0OheUXypTWtYfDfywvNbfbFo/E1gHbANe07T+EGBp6f+Wlu91Wln/feCwpvXHljfX88CxLdu8u2xzd/PYyDvfxo5kynhe+wTnGMBXgCfIBcE+RbzZ7R7Hm8s4vgzMaNN+oJntk8nRwE7gSeDIlrazyxhXDmtuJYO55X14Fp2L0b7Nhkne74wxt4uBV7VZfyb5D8ltwEvNreN2s8nv4c8DS6go4s1td/+bSp8PtY69tB/Y8vXA5DZhoQ/jAziu/EesavOfN4M8e7MZmN7rse7nHN5Tcvhk07rfK+sWtem/oLQ91LL+trL+kjbbvL+0XdO0LoAflPVz2mzzcGk7u9cZNY3pCvJs6BnAQtoX8UOfHflQv5Xl/TN7FP2HPrMyhlPKGO6taH8O2GhuCUYuRvsyG3q83xkptxG2XUzLhI+5te37BXIRfwSdi/ihz408270T+DotRwN0+J4Dk5vHxI/PgrJcnFLa1dyQUtoIPAq8CDh1sgc2yV4oyx1N6xrZ3N+m/8PAFuC0iDh4lNv8W0sfyG/enwGWp5RWjXKbninHzV0HfDyl9HCHrmaXZ0rmAPcBP418jPfVEXFFtD+u28yyFeTZzpMj4iXNDRFxBnnn8ZWm1eZWrV+zqfN+p92+Asxtt4i4GDgfuCyltH6E7uYGv02e9FkEzIyIt0bEuyPiDzqcRzAwuVnEj88JZbm8on1FWbY9qWIQRMQBwO+WL5vfEJXZpJR2kP86PYD81yoRMR14GbAppbS2zVO1y7I2+ZecbicfevSeEbqbHfxiWT4FfAv4F/IfQDcCSyPioYiY3dTfzICU0gbgauAo4HsR8ZmIuDYi7iLPgn4Z+MOmTcytWr9mU8s8I+JngXPIf/w83LTe3IqS0cfJs85fHKGvuWWNfcUs8mGqt5MPq/k0sDwiboqmk9AHLTeL+PGZVZbPVrQ31r94EsbSK9cBJwH3pZS+1LR+rNl0k2Wd8v8L4FXAxSmlrSP0NTs4siwvA6YBv0KeRT6JfJzhGeRjExvMrEgp3Qj8BrnAfBvwLvL5BWuAW1NK65q6m1u1fs2mdnmWTyvuJJ8cuDCl9NOmZnMDIl/hbRH5MIvLR7GJuWWNfcX7gW8CP0/eV5xDLurfDry3qf9A5WYRv39FWaaejmI/iYjLgavIZ1lfONbNy3Ks2Yylf1/kHxEnk2ffP5pS+o+J+JZlOcjZNWZOArggpfRASmlTSum7wJuAHwFnVhxa084wZJYHEfGn5KvR3Er+GHg68GryOQZ3RsT1Y/l2ZTnwuXWhX7PpqzzLLOjtwOnA35OvCtKNQc/tT8gn/76t5Y+c8Rr03Br7irXAm1JK3yn7igeBC8jnoL0zIg4a4/etRW4W8ePT+EtqVkX7zJZ+AyMi3kH+2O975JM5NrR0GWs2I/Vv91dt3+ffdBjNcvaeDejE7KCxE1uZUvrv5obySUbjU5+Ty9LMgIg4i3zps39KKb0zpbQypbQlpfQt8h8//wdcFRHHlU3MrVq/ZlObPEsBfwf5k6C7yJc3bS1ahj63iJgLfBC4JaV03yg3G/rcisa+4v7WT7nLvmMVeWa+cS33gcrNIn58Hi/LqmOa5pZl1TFRtRQRV5Kvw/odcgH/ZJtuldmUwnYO+eSmlQAppc3kAuPQiHhpm+/XLss65H8oeXzzgeebbtyRgPeVPn9T1t1Yvja7PeN7pqK98Yt7Wkv/Yc4MoHGjkq+2NqSUtpCv4DCFfGgXmFsn/ZpNLfIsGf0d8Bby9bR/p5xLsBdzA+DnyIcaXdK8jyj7iTNLnxVl3flgbk3GtK8YtNws4sensaM8N1ruWBoRM8gfH24FvjbZA9tfIuJq8o0Rvk0u4NdVdH2wLF/fpu0M8lnZS1NK20a5zRta+kA+3u2HwLyImDPKbSbbNvJNYto9/qv0eaR83TjUxuzyiW87gLkVH4OeVJary9LMssaVUmZXtDfWby9Lc6vWr9n0/X6nvGfvIc/A3wZcmFLa2WGTYc9tNdX7icYk2d3l69VN2w17bgAPlOVJrQ3lXIxGsby6qWlwchvP9Sl9JBiSmz2V1/Te8pq+CRw+Qt+Z5Lts9t2NUvrlQfV14s0uj+OOMo4PtKx/Hfk4x2eAF5vZXuP7rTKOJ4GXtbS9oeS2lXKnwGHOjdHd7Kkvs6GH+51R5HYw8K+lz82j+X81t47bLaH6OvFDnxtwELnI3gW8rqXtA2XbJYOa234JfZge7Htb3WvZc1vdx5ng2xH38HVeVF7TDvJM/MI2j4tbtjmfPbcsvxm4nqZbltPmxgzAR0t7862Qf1LWVd0K+dHS/g3y1XIm/XbuXWa6kDZFvNntHt+R5MtwJfLM/A3lte8oY3yzme0zvinky0gm8o2dFlGOkSfv5BJwxbDmVl7rreVxfxnPE03rbqhDNkzyfmcsuQG3lPangWtov684y9z2/Xmr+B5LqCjizW13/18iX7p0B/l9eQPwUNluHTBvUHObsNCH+QG8nPyLay35Y+ofkE/67DhbXacHewrOTo8lbbY7nXLDHvIM4P+Sz8Kf2uG5Lipvks3AxvJmPK9D/2nkncUK8qzZ0+WN/Mpe5zbKTPcp4s1u9/gOJ89arCrvrfXAvcCpZlY5vgOBK8kf0z5H3rGtI19r/9xhzm0Uv8dW1yUbJnG/M5bc2FN0dnosNLf2P29tvkcjz7ZFvLnt3uaV5KsfrSvjW0O+Vvwxg5xblCeRJEmSVBOe2CpJkiTVjEW8JEmSVDMW8ZIkSVLNWMRLkiRJNWMRL0mSJNWMRbwkSZJUMxbxkiRJUs1YxEuSJEk1YxEvSZIk1YxFvCRJklQzFvGSJElSzVjES5IkSTVjES9JkiTVjEW8JEmSVDMW8ZIkSVLNWMRLkiRJNWMRL0mSJNXM/wMrT+FIHyr5cAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 248.0,
       "width": 376.0
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses['train'], label='Training loss')\n",
    "plt.legend()\n",
    "_ = plt.ylim()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 显示测试Loss\n",
    "迭代次数再增加一些，下降的趋势会明显一些"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvIAAAHwCAYAAADEu4vaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdd3hUVfoH8O8JJdIFQVBAEEXFhoKKXcS6smLvq4L7U7Gs2NdFd0XXtaygKAKyIERBREEEAQUEQg0kJIQWEgIhnfTe25zfHymkTLl35taZ7+d5eDLM3PJOve899z3nCCkliIiIiIjIXoLMDoCIiIiIiNRjIk9EREREZENM5ImIiIiIbIiJPBERERGRDTGRJyIiIiKyISbyREREREQ2xESeiIiIiMiGmMgTEREREdkQE3kiIiIiIhtiIk9EREREZENM5ImIiIiIbIiJPBERERGRDbU3OwAjCSESAXQHkGRyKERERETk3wYDKJZSnqnXDgIqkQfQvVOnTr2GDRvWy+xAiIiIiMh/xcbGoqKiQtd9BFoinzRs2LBeUVFRZsdBRERERH5s5MiR2LNnT5Ke+2CNPBERERGRDTGRJyIiIiKyISbyREREREQ2xESeiIiIiMiGmMgTEREREdkQE3kiIiIiIhtiIk9EREREZEOBNo48ERER+RmHw4H8/HyUlJSgqqoKUkqzQyI/I4RAcHAwunXrhl69eiEoyBpt4ZpEIYS4XwgxQwixTQhRLISQQohFPmzvOiHEz0KIDCFEVcPf9UKIO7SIl4iIiPyDw+FAamoqcnJyUFlZySSedCGlRGVlJXJycpCamgqHw2F2SAC0a5F/B8BwAKUA0gCc5+2GhBDvAPg3gFwAqwFkAOgN4FIAowH85mOsRERE5Cfy8/NRXl6O9u3bo1+/fujSpYtlWkvJfzgcDpSVlSEzMxPl5eXIz89H7969zQ5Ls0T+FdQn8EcB3AAg1JuNCCEeQH0SvwHAvVLKklaPd/AxTiIiIvIjJSX1qUK/fv3QrVs3k6MhfxUUFNT0+UpLS0NJSYn/JPJSyqbEXQjh1TaEEEEAPgFQDuDR1kl8w35qvI2RiIiI/E9VVRUAoEuXLiZHQoGg8XPW+Lkzm5U6u14N4EwAywAUCCHGArgQQCWACCnlTjODIyIiIutprIlnOQ0ZobHB2ip9MayUyF/e8DcLwB4AFzV/UAixFcD9UsocTxsSQkS5eMjr2n0iIiIiCmzeVp7oxUqnr6c2/J0IoBOAmwF0Q32r/DoA1wNYak5oRERERETWYqUW+XYNfwXqW973Nfw/RghxD4B4ADcIIa7yVGYjpRzp7P6GlvoRWgVMJKW03Nk5ERERBQYrtcgXNPw91iyJBwBIKStQ3yoPAFcYGhWRE1W1dXh07i6MnroZ+9MKzQ6HiIjIlpKSkiCEwPjx480OxZaslMgfbvjrKitqTPQ7GRALkVtztx5DWEIekvPK8di8cLPDISKiACaEUPUvJCRE8xhCQkJ02za5ZqXSmq0AagEMFUJ0lFJWt3r8woa/SYZGReREdMqJ882SyloTIyEiokD37rvvtrlv+vTpKCoqwqRJk3DyySe3eOySSy4xKjTSmeGJfMOkTmcBqJFSJjTeL6XMFUL8COAxAP9C/WyxjevcAuA2AEUA1hobMREREZF1TZkypc19ISEhKCoqwssvv4zBgwcbHhMZQ5PSGiHE3UKIECFECIC3Gu6+qvE+IcTUZov3BxALYKOTTb2K+tlh3xZCbBVCTBVCLAXwO4A6AE9LKVmQTEREROSl8PBw3H///ejXrx86duyIgQMH4tlnn8Xx48fbLHvs2DE888wzOPvss9GpUyf06tULF110ESZOnIi8vDwAwOjRozFhwgQAwIQJE1qU8SQlJXkdZ0ZGBl544QUMHjwYHTt2RJ8+fXDvvfciKqrtKOPV1dX48ssvMWLECPTs2ROdO3fG4MGDcdddd2HDhg0tlt22bRvuvPNODBgwAMHBwejXrx+uvPJKvPfee17HahatWuQvAfBkq/uGNPwDgGQAr3vaiJQyWwgxCvWt8fcAuBJACYA1AD6SUu7SKF4iIiKigLNgwQI8/fTTCA4Oxrhx4zBw4EAcOXIE8+bNw6pVq7Br1y6cccYZAOoT6csvvxzFxcW44447cN9996GyshKJiYlYuHAhXnzxRZxyyikYP348Tj75ZKxcuRJ33XVXi9Kd1mU9SiUmJuLaa6/F8ePHMWbMGDzyyCNITU3F0qVLsWbNGvz888/485//3LT8+PHj8cMPP+DCCy/EE088gU6dOuH48ePYvn071q5di5tvvhkAsHbtWowdOxbdu3fHuHHj0L9/f+Tn5yM2NhazZs1yWqZkZZok8lLKKQCmKFw2CfVDTLp6PB/1LfOvahAakS6sMZ8bERGRcvHx8Xj22WcxePBgbNmyBf379296bNOmTbjlllswadIk/PLLLwCAZcuWIT8/H9OnT8ekSZNabKusrKxpNt3GEWdWrlyJu+++W5MRaCZOnIjjx4/jgw8+wNtvv910//PPP4/rr78eTz75JJKTk9G1a1cUFRVhyZIlGDlyJMLDw9GuXbsW22q8cgAAc+fOhcPhwObNmzF8+PAWy+Xm5voct9Gs1NmViIiISFOD31pjdgiKJX08Vtftz549GzU1Nfjiiy9aJPEAMGbMGIwbNw6rVq1CSUkJunXr1vRYp05tBwzs0qWLbnGmpaVh/fr1OOOMM/Dmm2+2eOzqq6/GI488gkWLFmH58uV44oknIISAlBLBwcFNJxfNnXLKKW3uc/acevfurd2TMAgTeSIvcAooIiKym5076+fT3LJlC3bv3t3m8ezsbNTV1SE+Ph4jR47EuHHjMHnyZLzwwgtYt24dbrvtNlxzzTU4//zzdZ0MMTo6GgBw3XXXoUOHDm0eHzNmDBYtWoTo6Gg88cQT6N69O+68806sWrUKl1xyCe677z5cd911GDVqFDp37txi3cceewzLly/HqFGj8NBDD+HGG2/ENddcgwEDBuj2fPTERJ6IiIgoADSWmHz66adulystLQUADBo0CBEREZgyZQrWrl2L5cuXAwAGDhyI119/HS+99JIucRYVFQEATjvtNKePN95fWHhi/JMff/wRn3zyCRYvXtxU537SSSfh/vvvx9SpU9G3b18AwL333ovVq1dj2rRpmD9/PubMmQMAGDlyJD766CPccsstujwnvTCRJyIiIr+ld7mKnfTo0QNAfaLcvXt3ResMGzYMP/74I2pra7Fv3z5s2LABM2bMwKRJk9ClSxf89a9/1S3OzMxMp49nZGS0WA6oL5WZMmUKpkyZgtTUVGzduhUhISFYtGgRkpKSsG3btqZlx44di7Fjx6KsrAzh4eFYvXo1Zs+ejT//+c+Ijo7G+eefr/lz0ouVZnYlIiIiIp1ceeWVANAiqVWqffv2GDlyJP7+97/jhx9+AACsWLGi6fHGDqZ1dXU+x3nppZcCALZv347a2raTLoaGhgIARowY4XT9gQMH4rHHHsO6deswdOhQbN++vUWH10ZdunTBmDFj8Nlnn2Hy5Mmorq7G77//7nP8RmIiT0RERBQAXnzxRXTo0AGvvPIK4uPj2zxeXV3dIsmPiIhAVlZWm+Ua72tef97YoTQlJcXnOAcMGIBbbrkFSUlJmD59eovHwsPDsXjxYvTs2RP33HMPACAnJwfh4eFttlNWVoaSkhK0b98eHTt2BABs3LgRFRUVip6THbC0hoiIiCgAnHfeeZg/fz6eeuopXHDBBbj99ttxzjnnoKamBikpKdi2bRv69OmDuLg4AMDixYsxc+ZM3HDDDTj77LPRs2dPJCQkYNWqVQgODsbLL7/ctO2rrroKnTt3xvTp05Gfn99Uk/63v/2tRQmMUl9//TWuueYavPHGG1i/fj0uu+yypnHkg4KCsGDBgqaRddLT03HllVdi2LBhGDFiBAYOHIji4mKsXr0amZmZeOmll5qWfe2115CUlITRo0c3TTQVFRWFTZs2YdCgQXj44Yd9fZkNxUSeiIiIKED85S9/wfDhwzFt2jSEhoZi/fr16NKlC04//XTcf//9eOihh5qWfeSRR1BVVYWwsDDs2bMHFRUV6N+/Px5++GG89tpruPDCC5uW7dmzJ37++We89957WLBgAcrKypr2500iP2TIEERGRuKDDz7Ab7/9hs2bN6N79+64/fbb8fbbb+Pyyy9vWnbw4MF47733sHnzZoSGhiI3Nxe9evXCueeei48//rhFcj558mT88ssviIyMxIYNGxAUFIQzzjgDkydPxssvv4yePXt687KaRkgZOFPbCCGiRowYMcLZ1L5Eavw1ZDc2xmU3/Z+dqYiIzBEbGwugvlMmkRGUfuZGjhyJPXv27JFSjtQrFtbIExERERHZEBN5Ii8EznUsIiIisiom8kRERERENsREnsgL+k1MTURERKQME3myJYdD4s1l+3Df7DDEZhSbHQ4RERGR4ZjIky0tj07HT5FpiEouwJPzI8wOh4iIiMhwTOTJlsIScptuZ5dUGb5/dnYlIiIKPFYbtp2JPBEREdmWEPW9lhwOh8mRUCBoTOQbP3dmYyJP5AVrfH2JiCg4OBgAmmYSJdJT4+es8XNnNibyREREZFvdunUDAGRmZqKkpAQOh8Ny5Q9kb1JKOBwOlJSUIDMzE8CJz53Z2psdABEREZG3evXqhbKyMpSXlyMtLc3scCgAdO7cGb169TI7DABM5Im8wrYeIiJrCAoKwsCBA5Gfn4+SkhJUVVWxRZ40J4RAcHAwunXrhl69eiEoyBpFLUzkiYiIyNaCgoLQu3dv9O7d2+xQiAxljdMJIiIiIiJShYk82ZPJV005ag0RERGZjYk8EREREZENMZEne2KTOBEREQU4JvJEXuB4CERERGQ2JvJERERERDbERJ5MV1Vbh4W7krEiOl352L/s7EpEREQBjuPIk+kW7kzGB2tiAQBdg9vj5vP7mhwRERERkfWxRT6AOBzWrOxuTOIB4P3Vh5StxCZxIiIiCnBM5APEFxuOYPj76zFr81GzQ/EL1jwlIiIiokDCRD4A1NQ58PmGeJRU1uK/aw+rXj+toByTfzmAHyJSdIjOOtIKypFeWGF2GERERESKsEY+ANT5WFLz0g/R2JNSCAC44PTuuHjAyVqE5RuNm8SjkgvwwNdhAIAVL1xjjedIRERE5AZb5MmjxiQeAH4/mGliJPp55rtIOCTgkMDEhVEel2eJPhEREZmNiTzZk8aZdF5ZddPt7JIqbTdOREREpAMm8qSK3i3RwiZN3ezsSkRERGZjIk+WonQ+KD0zaSbpREREZAdM5ImIiIiIbIiJPKmid+mLFUprLBACERERkUdM5MmedMy2lZTWMNknIiIiszGRJ0thgkxERESkDBN5shTFHU1N7pHKDrFERERkNibyRK14c1Ugt5RjzxMREZGxmMiTKkLn4hcrlNZ409p+LKdM8ziIiIiI3GEiT/ZkcsZvhRMOIiIiCmxM5Ik0IBXPZEVERESkjfZmB0D2YoVx3gFo1ttUSokXf4g2a/dEREREXmOLPAW0TXHZWLM/w+ftCMuc4RAREVGgYCJPAY2dVImIiMiumMiTPbEBnIiIiAIcE/kA0Lofpi8dMy2TP5tcpG6Z14GIiIgCFhN5Ii+wsysRERGZjYk8WYoVOo16c8WCw08SERGR0ZjIExERERHZEBP5ACBbFYL4ReOxjg333lwVsMKVBCIiIgosTORJHaskrBbr7MrSGiIiIjIaE3nSXWVNHUqras0OQzElSTnTdiIiIjIbE/kAZGQSmppfjis/2ogrP9yIg+lFBu6ZiIiIyL8xkQ8AWlZ9qC2seXPZfhSW16C0qhZPhezWLhAdsUaeiIiI7ICJPOkqPquk6XZ2SZV2G9Yxb+bwk0RERGQHTORtwOGQ2Bqfg4jEfL9PGBXn5/79MhARERF5xETeBtYfysQT8yPw4JydiEwuUL1+65zXyicDSiMz+xm0PuFgaQ0REREZjYm8BR06Xoyj2SdKUiYu2tN0+8XFe5ytQgaz08kRERER+Scm8haz7UgO7vhyG27+bCv2pRa2ebzOYW7CqHfDs7ebP5BWPyJOYxlS+LE8JtdERETk15jIW8zj30Q03X7++7at78xNnbtn1g7klVZhXUx9GdJD/9vlVRkSERERkV0wkbewksoaTbbTumXaH84FWj+nWofE9+EpeO57liERERFRYGhvdgCkn6KKGrz3awzKqrWbVVWoLH4x8qSh9dUKPcuQ2LWViIiIzKZJi7wQ4n4hxAwhxDYhRLEQQgohFmmw3ccbtiWFEP+nRayB5L9r47A8Oh3rYrIUr1PnkPj8j3i8u/IgCsurdYzOs4jEfNw1cwc+XRenaHnZ5rTBu3RbSf7vD1c1iIiIyN60apF/B8BwAKUA0gCc5+sGhRADAcxo2GZXX7cXiL4PT1G9ztLIVHyx8QgAoKKmDv+9f7jWYbnXLPd+cM5OAMC+1ELcPKwvLj2jp2FhZBRV4LQenQzbHxEREZFaWtXIvwLgHADdATzn68ZE/aDcCwDkAfja1+1RS+46zIaEJTXd/ikyrc3jn2+IV9xC7hUXsR3KKNZvn04cL6w0dH9EREREammSyEspQ6WUR6R24/29BGAMgAkAyjTaJimgZGKjmaEJOJZTakA0J/xzxcEW/3f2QWv96eMcTUREROTPLDdqjRBiGICPAXwhpdxqdjxWo3dtdpDC5De9sEKfAFzs35t+q8zjiYiIyJ9ZatQaIUR7AAsBpACY7MN2olw85HPtvj9o2yn0BKWt2EpHr/H2Is3SyFSv1tMKW/OJiIjI6qzWIv8vAJcCGC+l1KnJ197yy6rx0e+xus1aGqQwg9Uz0T2WU4o3lu1XvR5HkiEiIqJAYpkWeSHEFahvhZ8mpdzpy7aklCNd7CMKwAhftm0Fc7Ycw4Wn98Cdw0/XfNtKauT1tvNYnsdllJzHND4VKSXWxWQir6wa940YgJM6tPMxQiIiIiLzWSKRb1ZSEw/gnyaHYwvrD2XpksgrrZG3k/DEfExcVD/La3FFLZ4bfZbJERERERH5ziqlNV1RP3zlMACVzSaBkgDebVhmbsN9002L0k+k5lcgKrnAaXmO4tIarYPSYrutnk9jHf8Haw413ffJWh2HziQiIiIykCVa5AFUAfjGxWMjUF83vx3AYQA+ld34C3c18nUOiQ9/i3X5+M2fbQEAfHr/xXjgsoEtHjuudDQanTJ5dxUzUsqm0h+96+H98MIEERER+RnDE3khRAcAZwGokVImAEBDx9b/c7H8FNQn8t9KKecZFacemieielqyOwXfbE/0uNwby/a3SeQzipRNhKR41BpFSynbtpTuO9m62pfSWImIiIjsRJNEXghxN4C7G/7br+HvVUKIkIbbuVLK1xtu9wcQCyAZwGAt9m8H76w4gN8OZOKdscNw74gBPm/P3QnBiuh0n7dvFl9S7l2tOslaoN8uERERkW60apG/BMCTre4b0vAPqE/aX0eAOpJVgkW7UgAAr/60T5NEXq/hJ5VqnSSn5pfjj0NZuOX8vhjYq7NP265z8dya3+vs+e9OKvBpv81ZYfQeIiIiInc06ewqpZwipRRu/g1utmxS6/sUbtu2ZTWZxcrKVeykeZorpcSTCyLw/upDmBCy2+O6tXUOrI/JRHSK88R7fUym0/vNPnlxx7qRERERkb+ySmdX8kJZVS0W7kpG767BuG9Ef9NakaUEjuWUAQCOZpd6XP7HyFS8/ctBAMAfr1zf5vFtR3Kd70dlXFq8GuXVtSitrMWp3U/SYGtERERE2mEib2NfbjqCOVuOAQBO7RaM68/pY0ocrRNsT516G5N4AHh7xUGXy7XZj8pM3tcTm7zSKoyZtgVFFTWYcM1gvHnbeejU0flkUizEISIiIqNZZRx58kJjEg8AMzYdMXTfzZPk1iUvahLuiMR8r/avJqf3Np//6Pc4FFXUAAAW7EjCY/N2ebchIiIiIh0wkfcTRpePN0+O3Y/9rt0+pcGV6BlFLcfU35NSiMLyakNjICIiInKFibwB/HEc8+bPqLSytsVjeqXbRp+sONufhfvbEhERUYBhIu+HjE42P/49rtX+9QkgS+XoP772/WXSTkRERFbGRN7CiitrUVReY3YYTjVPkn+MTG3xmC/5r7sOqm/9fMCrnXiTz6tdhzk/ERERGY2JvAF8aRn+ZF2c54V82N+EBRGornW4fPydFQdcPKJsJ1q2zu9sNXOrYl6+AWpq8tMKKjwvRERERKQhJvIW4iyhXhyeomjdlrOeKt9n6OEcfLM90eXjjTPSqmGVkhRX+bu7E5fm1DyP15fuU74wERERkQaYyJvA4ZDYnZSPsqqWnUR/iFCfNGshMsm7ISBdMXp0GbXmbjsxbKerZF8IlstYgZQSUcn5OJheZHYoRERElsNE3gTv/hqDB77eibFfboPDIVvcr5TZSaZJk8g28eVk4dN1h09sR/UkU17vlrywKS4b983eiT/P2I6oZG1POImIiOyOibwJFu5KBgAk5ZVjV6KXdd+tRCUXICGnVJNtKeEunzWitEbJPnwZ9vOR/+3yerIq0s5fv41suv3i4mgTIyEiIrIeJvIma6zXVjs6jbMU9dG5xs08KoRATkkVSluVBxm3f323X1Zdp+8OSLWKGr4nREREzbU3O4BA4Lb1GkBeaRWu+2+oqm06a5DOKq5StY3WKlUkSmEJufhsfTxO6tCuzWPNW8vNLAFqTPa1zPn9cXIvIiIisicm8hYwdX08yi3QAvz1lgTFy/53bX2dudYt8krTZLNGxkkvrMCWIzkorzbnSgQRERFRIybyFlBQVq16na3xOZrHcSBNm5FBmndE1WtiJbMS+btn7kB1nbLhK4mIiIj0xBp5s0nv6r1LKrVvEdaq7rx5ku0wsbZGjyIYJvFERERkFUzkLcA6QxpqE0jz3F1t6Y2m9ewNL6x1Xl8iIiIi7TCRpyZMeImIiIjsg4m8ySSkpiOhhB7O9npdraKQBhSwazV7LE9eiIiIyK7Y2dVkWue8Exbs9mq9xNwyrD+UpUkMZs8620i0+uuMWZ1mrWh/WiGW7E7FuOGn48ohp5gdDhEREXnAFnkjuMkkl0amqRq/XS/PLYoyOwQAQGWtMa/FTdM2Y+3BTEP2ZRfjvtqBxeEpePh/u1DDTr1ERESWx0TeZGtjMrExzvtyGC3EZ5cgLrNEs+350sqdml+hWRzuJOSUYaJFTl6sqLhC3UzDREREZDwm8qR98myVchXWv5MGauschvT7ICIiUouJvAG07MxqB1p1RHVm+ob4+n0o2EVTjTx7tJKX4jKLce0nobj5sy3I92LiNiIiIj0xkSdbmb7hSH0LqdmB+Dm+vvWe/i4SmcWVSMgpw/urYswOh4iIqAUm8qQ5vasQtNw8G+vJneZlZ3tTC02MhIiIqC0m8qQ5vVtzHVLiDwVDZSopqWHps3M8vyEiIrI+JvKkOb07Bq6MPq5qeSalRERE5I+YyJPtfLM9UdFyShL4w1naDbtJREREZCQm8gYItDpsd+3xb/283+fta/l6LotK025jRERERAZiIk+ac1dZs2R3qmFxBNoJFBEREQUWJvLk95jQExERkT9iIk+a03NCKCtLKyg3OwTSEScWIyIiq2EiT9qzSB5v9Iy6m+KyDd1foNHi3Swqr8GcLQnYGp+jel29R2MiIiJSq73ZAZD/0TvdictUN9KMUQk98zx9afHyvr/6EH7eU9/BecsbozHolC4abJWIiMgcbJE3AC/Im8PoSgh/arH1n2fSUmMSDwCLI1JMjISIiMh3TORJc36Uz6oSoE8bQP1JzIG0IuSUVJkdinIq3zDWyBMRkdWwtIY055ASSyNTUVRRY3YoAICqOoch+3lv1SHccE4fDOnT1ZD9WcmS3an4x/IDOKlDEHb8fQxO6Rqs+T7MTqP96YoLERH5B7bIk+Y2xmXjjWX78cGaWLNDAQDsSy00bF/PLdpj2L6s5B/LDwAAKmsc+OyPeF32wTSaiIioJSbypLlPfo8zO4QmNQa1xjc6nKWuI64/Kq2qNTsERdSeGLC0hoiIrIaJvAECLQGwUiLnYDkEERER+Skm8uS3hBDYdSzf7DBII4F1OkxEROQZE3nyW7EZxXhyfoTZYQQcuyTc7LxKRER2x0SeiDSlVykZ024iIqKWmMgTEVuniYiIbIiJPJHGqmuNHSlHD1bM65u38x/NLsVLP0Tju51JJkVDRERkPibyBtiTUmB2CGSgpVGpZofg954K2Y1f9x3Hv1bG4GB6kVfbsOLJSqCqqK7DL9FpOJzJ4VuJiNRgIm+Ajy00rjrpr7DcGjPaqtE6qZU+VKTr1dm1eUQp+eVNt7cdyfV5e2Suz/44jFd+3IdxX21HkQ2/P0REZmEiT0REppq7LREAUFXrwMJdSeYGQ0RkI0zkicgWrdN2GdaSiIjIKEzkiTTmFyPA+PIUbJJxq32bbPK0PNoUl4V/rz6E1GblSUREZE/tzQ6AiMznFycfOvOHVyiruBJPhUQCAMIS8vD7pOtMjoiIiHzBFnkiasNOSatO80/5pR1HT3QMjs0oNjESIiLSAhN5ItI0cRcGF6HwYoJx0gsrMDP0qNdDfhIRkbaYyBNpzB8SS394Dlpjwz/w7MJIfLruMMZ9tR2VNXVmh0NEFPCYyBORrRN3b0tr1I6Vb+OXSDMH0+vLcRwSOJJVanI0RETERJ6I2vBpQig2XRMRERmCiTyRxuzYcutL4m5XgTr8pFZ4wkZEZD4m8kQas3OZSiN/eA5aO5ZbZnYIRIaQUiKvtMrsMIhIASbyRKQpNtQS2duTC3Zj5Acb8NWmI2aHQkQeMJEnojYt8GyQJwpMMceLsDU+BwAwdX285tsvr67VfJtEgYyJPBERWYZg8b2pSir1S7Q/+i0WF767DlN+jdFtH0SBhok8EbUhbVgkn5pfjvELIvDmsn2oqXOYHQ55yY6fPauprKnDxIVReHDOTqTklataV8+Xf87WY3BIICQsCVW1nIeASAtM5Ik0ZqURYKJTCvDswkgs35Nm2D6NblBt3N2kJdHYfDgHP0Wm4budyR7XC8SEkY3dgbCL7nAAACAASURBVGFW6FGsjclERGI+XloSbXY4TgXg149IF0zkiWwup6QK09YfxtqDGW0eu2dWGNbFZOHVn/Yhu6TS5Tba1Mjb8CC7J6Ww6fa6mMwWj1XW1GHyLweMDonIFBtis5tu700tdLMkEdlde7MDIPI3RifBb/9yAOsPZQEA/njlegzt283pckezS3Fqt5OMDM08rd6D2ZsTsDg8xZxYiIiIdMIWeSKba0ziAWCZlyU0WpYDCTcDUB7OLEFFtfG1sYsj2ibxNrzoYCks0/FPVioNJCLPNEnkhRD3CyFmCCG2CSGKhRBSCLFI5TZOEUL8nxDiFyHEUSFEhRCiSAixXQjxVyEETzqINPJTZCoe/yYcYQm5hu1zZuhR3DZ9K27+bAs7o5LlFFfWBGS/CQpcP0Wm4rP1h1FYXm12KOQDrUpr3gEwHEApgDQA53mxjQcAzAaQASAUQAqAvgDuBTAPwJ+EEA9I/tKSxVn9A5pXWoU3l+0HAGw7koukj8caUiP/6brDAID0wgr8uvc47hs5oOmxiuo6dOrYTvudku2YMfzkkogUvLPiIC4e0APLJl6NoCBebiD/FnY0t+k4kFVchU/uv9jkiMhbWrVyvwLgHADdATzn5TbiAYwDMEBK+ZiU8h9SyqdQf1KQCuA+1Cf1RLZ3ML0Ij87dhWnrDxu+77SCCl23ryQPK2s2KcyUX2Nw4ZR1+Pj3OB2jamtLw6Q3ZC1mtNW8tfwAah0Se1IKW5SqEfmrb7YnNt3+MTLVxEjIV5ok8lLKUCnlEV9ay6WUm6SUq6SUjlb3ZwL4uuG/o30Ik8gyHv7fLoQl5GHGpqOGlrcAyq4YGFUnW1Vbh5CwJNQ5JL7ekmDIPhslqxxfm1py1xfCznJKq8wOwWc+XdSw+iVFasHbtItvs/+wS915TcNfzu1M1qfgh7W06sRHOTKpQM9oFDHrR73OoX7P2cWuh9FsxA579sWZXX3HAtTAEHO8CKOnbsaDX+80ZRABsgbLDz8phGgP4ImG/65VuE6Ui4e8qd0n0pUVD7q+xKQmD6upU7ej91bFYMGOJI/L7U4qQE2dAx3a2aWtghqxGxSRMuMX7EZOSRWS88rxVegRvHEbU5xAZIej3McALgTwm5RyndnBEPkjM5Kn7OJKXPvJJlXrOEviXZ04LIs6MRQn23gpkPCiRmDIKTlRBhaVbP6VXTKHpVvkhRAvAXgNQByAx5WuJ6Uc6WJ7UQBGaBMdkT0pKU3Rm5TAP1ceREmlftVyH66JxSNXnFG/P932Erj0ShZZWkOkHi9kBS7LJvJCiBcAfAHgEICbpJT5JodEpIiVf0/nbTuGD9bEtrm/dcxGPIcjWaWqlleb4Fn5fSAiMhNL2PyHJUtrhBAvA/gKwEEANzaMXENEPhAQTpN4PfbkcQkBlFSpa43ngcd3dhtpRkqJxNwyQ997e71C2uO3jMheLJfICyH+DuBzAHtRn8RnmxwSkd9rnSdtP5KD6lp9Z19tXt9J9mNEBcyrP+3DjVM348UfovXfWQMmsvrgiTiRPgxP5IUQHYQQ5wkhznLy2D9R37k1CvXlNMYOsE2kATOPV1lFldh8OBu1db4l4f9cGYPXlu7TKCrzBXorq139Ep0OAFizPwM1Pn6miYj8kSY18kKIuwHc3fDffg1/rxJChDTczpVSvt5wuz+AWADJAAY328aTAN4HUAdgG4CXnNTEJkkpQ1rfSUT1Vuw9jhV7j+PFG8/G67edq2idksoa/HPFwTb3r9p3HDMeuVR1DEpaar052WEnSDICP2XGMKvBo84h0S6I7zL5D606u14C4MlW9w1p+AfUJ+2vw70zG/62A/Cyi2W2AAjxIj6igPJV6FHFifz0DUfw677jOkekjJTSZcLOS/NE+vPnr9mn6+Iwf3sSnht9Fl66aajZ4fitqto6pBdUYEifrmaHEhA0Ka2RUk6RUgo3/wY3Wzap9X0KtyGklKO1iJdIT3rNKrpoVzIm/3IA6YUVmm73m+2Jmm6vuaPZpXhh8R7M23ZMt324ojTxDzuai8e/CcePu1M8LhtzvMjXsIgCktknCDV1DswMTUBFTR0++yPe3GAsQK+3o7bOgVs/34ox07Zg9uYEnfZCzVmusysRtRWVXIB3VhzE4vAUvPLjXrPDcat5e/pfv92NNfszFI+W4+5gX1Be41tgLjw6LxzbjuTi7z8fQF6p+w64d321Q5cYiEhfdQ4/vtRgIWsOZCA5rxwA8MnaOJOjCQxM5Il0Ul3rwIG0Ijg8HECUtOCvalb6EpFonykVGn/QlfLmUKvlkIqernbUMhloYrehLAOJHbqT6HXlksxVrOMkf+QcE3kiHUgp8dD/duLOr7bjH8sPmB2OoZQkESFhSZrtjwkBkXb0+j6Z/S01u7SHSC9M5Ik0JiWQlFeO6JRCAMCPkamK180rrcLPUWltxli3QwubGom5ZU7vr3NI5Hoob1HDU53835ftb/F/tjL7h/UxmXhwzk4s35NmdihEhuB5SuBiIk+kA3djXrur1XxmYRReW7oPT4XsDshRWv48Yxuu+M8GLNqVrHgdAYGK6ro295dV1+HGqZtxIM11B9XWJ1n+3rqv5Qmh3ieXvnz+n1kYhYjEfLz60z5U1rT9bJD5jP5587fGEF8F4OHFbzGRJ9KBq2OGlBJ3zdzucr2o5AIAwIH0IlQ1m1nVrJbi5DznLed6ic8qhUMC7zgZ196d5dHOW16T8srx6Nxdmh7E7XyCZePQvVZWxZpdKzD7exOIn31T8IU2HBN5Io25+xkLT8zHwfRiRduxQgvSv1bGmB2CIu6ucpRU1So+tiyNTMPPUWlOW/gb8ThljNavs7evO98udfj5JruLSi7AdzuTUFypz0hnVqPVhFBE1EBK10l4hZPL/I0HzqKKGqf3a+Xj35UNAdmcN62ZZlw96BqszU/Zwl3JWLgrGYezSjD5jmFOl2GeQ+Q7fo/8lMktUNkllbhvdhgAIDajGB/de7Gp8RiBLfJEunD+YxbczvVX7j9rDrnemga/jfvc1IpbUamKk4gObl5XQP3r97+triewMrtEwCoscMGIbMTsb40VrnAGBJN/H5dEnOj39EOE8oEm7IyJPJGBOrZ3/ZX7KdL6I2y468SrtRkbjxi2LzXMTkh8YadkpvXr7G1HZLV5hZ1eI1KO59/kr5jIE+mgdTKQX1aN8upa7LdZq3jzY9/UdYdx4bvr8Nn6w27X0SoRmuOmVZxamrb+MK7+aCOHW3RCzZUdgAkfBQZ+zP0HE3kiA1z54UaM+nAj3l/dtnxGSeJrdiOhwyHxVehRVNU68OWmoyZH05IQxraiWi3RyyyqxIxNR3G8qBKv/rTP7HAsZ5qHE8/W8kqrdYoksLXtvGzsF8nvr7RY7HeJjMNEnkhjErJN4l1d50CJi6mrXR3PfD3OxWeV+LaBZtSE4i/Hy/TCCqf3W22seS0n0FLDyuPIN7d6f4aq5T/fEK/Jfu3KWp9u7VjtBJxIK0zkiXQgVGY52SWVbe7zNWG89fOtPq0PeJdMbYzL9nm/VnDtJ5uc3s+EwBx83YmI2mIiT6Q1LxKOf61wPV67wyERlVLgQ0DGSiuob8kuKrf3GL7+kDiGxmVjwY5E1XXiVuAHL79pzJpAzp3WDRN8f83lr6Nv+enTcovjyBPpQO1hdG1MZpv79qYU4uqze+Pfaw4hOqVQm8AM9M+V6mZntQu7HChiM4oxIWQ3AOB4YQXeHnu+yRFpK7+sGj06dUC7IOslrURERmGLPJFFPTovHBXVdViwI8nsUNq03ihpzfl133G9wjGV1WrkXZm1OaHp9txtiSZGor1lUWm4/D8bcNv0rYYOiRoI/LWllgKD33dqdoKJPJHGdiTkqvoxcXfY1LLDqjfsckg3spTALnmO/gmZvq+5u/BfX7oPdQ6Jo9ml+CkyMCZ9sTt37+eBtCIs3JnUZnZrIrXs8vusJZbWEGnsYHqxqsRyx9FcHaPxXU2do82zkdJaLR92aSUn7WUXtxy1p7zafv0BAlFjwlVYXo1xM7dDyvrZp6c+MFyf/bWu0ZdS9aAERFbEFnkikxWUW3fc6uiUQgx/bz2e/i7S7FB8omWLvR1OGRwOicTcMrPD8EnbxKvxr+t3YEt8Di77YIOeYZHGVkSnN723y6I4oZm32JgRuJjIE+nAnxp6yqvrEHo4x+ww3PKUqGt5kLNDDfEzCyMRc7xY0236+rzXxWTiv2vjkF3cdqhVNaa6meDpyfkRKK+u82n7gc7oT7f1v01kJ/507FWKiTwRqRbIB1+znnvo4Ww8MT8Cqzx0Ii6qqMGGWGuN5Z+YW4ZnF0Zh1uYEvL5sv0/bmhma4HkhP+RwSLy5bB/unx2Gw5nm9p3RRNMVFoN25+c/WlYcctQM/v4+O8NEnohsLSQsCQk5pW6X0bS0xqQDxYQFu7E1Pgd/+yEalTWuW531GsWl7fNW/kI0P/nYGq/s6k4gHpDd+XlPGn6KTENkcgHGL4ho8VhuaRU7iqrkb58vq5TWWCOKwMJEnkgHK6LTlS/s5pfPqj+KViovSSuowGd/xBu3Qws89TI3Ezw5dHpvfNlqfpnv/UBcJSoWeDsMEZaQ13Q7o+hEeVJUcj6u+mgjRn24wdL9IvT4WEopEZGYjywvyrUOpBdpHxCZjqU1RKSJaSoSy2MWPvi6smR3qulDY5rFKi1fdvH+qkMICUsyOwxVSirt07o9fv5u1NRJVNY48OpPe33foMEfb192N3fbMTw4Zydu+DQUhR4GDWi9n5d/1OC1ojbMzqMt1MZkGCbyRKTaOysO4r7ZYbYZ6i/Txw6Wzak5UOxJKcA/lh9ARGK+Zvv3SKcDWdtJwZStN3+HdpNRZZdo9z6689HvcYbsRwslza7OpBVUmBiJOlqcEH/4W/37VFnjwOzN6vpOlLq5qhUI9Ep4AzCPNh0TeSLySkllLdbFZDp9zMqX+H2l5kB176ww/BCRggfn7ESdw5hDnL8eSKUE3l0ZY8i+FoenGLKfQOKyNMrLjDI5r+VvjLt+I87369VuiSyHE0IR2dAfh7JQU+dAYbm5JQCuOpHeOHWzsYHYQFVtHTp31P8n1x9a2g6kFWHrkbadYrdbfPI0PalJeAOhTjgqucDsEMiJAPjoWQ4TeSIbsvsETXbmbQti89WWRaUh/Fgenht9Fob06apRZA37sUibfHphBapUtpIC9SUPd3613eljQc4yVDat2pLRb1vb7y0/N3rgq2o8JvJEFpZWUG52CG5ZJWk0kq/POCGnFK8v3QcAiEjKx5Y3blS9DXdTy+vWIt9qu+52E5tRjLFfboM31UShcc7HwJcAgtjcRw1afwUC75fIN4H42+2vWCNPZGEvLo42OwRqxVWiXFvnwJb4HOSVVrldf0ez8pDkPO1P1KxweH55yV6vknhPnLbIK/DZ+sP+MYmSjbn63nh74pluo469auSWVuHZhZGYtCTa7TCzrfHCVOBiIk9EpIKrlqwPf4vDk/MjcNv0raiqVV9S4nU8rVvKdRtHXvl288rcn8x4y92VCHe+3HQU98zaYan5D7Sw9qDzzua+sFJL7cbYLDwxPwK/H8hocf9Hv8di6np1c0e0KayxztNs4d1fY7AuJgsr9x7HFxuPNN0vpcS+1EIU22ho1OaklAg7mov1MZmGdfwPFEzkiYjUcHEMahxmMbe0GmsPZmJ3koFDTjbjLkGpqWv54K5jeS6W9G0/enV586W0pry6TperBGY5klWCiYuizA7Da0reir9+G4mt8Tl47vs9qG02Y/GcLcf0C8xka/afOGlZviet6faMTUdx18wdGDN1s+oReqwgKrkAj84LxzMLo7B6/3HPK5BiTOSJyGsx6cXYEt92dBF/piQBOZhehAe+3un0MbPKvNfHZDbV5jdq3dLpjtktmFJKp6U1fpSbu9X6asSyZkmeHbh6n1xdAShqNSJXdbNEXs94rKpx9urc0mosjbLXew+0nIBr0hJOxqUlJvJE5LV52xPx5PwIs8Mw1D+WH0CNh6Ri7jbtJkFSy1XC/czCtq23eiUzvgx/6K58JpA7u7YpC7JYJlrnkMjSaOK1tQczcfl/NrS4z9cTSaNK0IxQWW2/FnmHSZfDtjkZxtbfMJEnIlJhU1w2vg1L0mXbVbV1mLMlAf/bmoDqWmUtkG1H7zBq4inX+9Er33aW5EupPCmzc/Kmhi+vvzcvkcMhcdfM7Rj14UbM3aqs7MXdezFxUZTmLfBkLrO+eXEZ/t/JnYk8EZFKq/crL0lp1HQgc9PivGhXCj76PQ4f/haH78OTlW23TUuj6tC82o/RJJy/dF+FHsWdX21HYXm14TFRvc3x2TiYXgwA+M9vsU6XcZW4K/1cBcYpmHHM/j6rsS4mE/fM2uHVjMtW6rytFybyREQq6dWy++/Vh5puv9/sthpqItPrYO5TaY2bx1wNP3kwvRgfukggm3v31xgczfb/FjqjmT3DtCL+n8/5rWcXRiE6pRCTfzlg21F79MREnohIpeY5QWVNHTKLtKkNbrEP6bozqrtk16jhJ33ZTV5pFdIL1Y8D7q5GPiLR8yhB34en4LF54U3/9+a1WrgrGbdP34pfovXpcBiVXIDZmxOQU+L9EJ5m56zPfx/lcT4F2eqv0QrKa/yq1Mrqz0Srl7p1J2jizK5ERKo5Go5KZVW1GD11s8ekBfAuaXzu+z1I+nisqnWscEAXHqq0r/poE2odDiz66yhcfXZvRduU0v2EUEqfd1Zx/XsVl1mMZxdGoWfnjgrXrD9p++eKgwCAV37ch3suHaB4XSUKy6tx3+wwAEBEYh4WTLhC0+3rISWvHGWtOl/+diATHdsFYfrDlzbd5+vnUo+k+49DWbj1gn6ab5fISGyRJyJSqTGnaGw5VTMgg10HXlGTR3kqramuc8Ahgce+CW/z2NGcUq+3q1ROSRXGzdiB5Lxy7E0tVLxehc6jhWyKy266HXrY/Wgb7t4Ooz5jK6LTcf2noU0nNy0e22utscKd1Uo7G8nJrtS+50ZfjAiEWnWzMJEnIlKp8SCYXaJ9SY2vzKgWqKlzYHdSvuoZbZ3F6m60Hm9ndm3t6o83ajIqSnFlDT5dF4dvtieiqML3S/56vXdKrhh5E0PzscG95UfVLeSGu/f556g0vLF0HxKcnMSXV9fqGJV/YGkNEYCxF52GNSomx6HA1lha46qEpH2QQK2G4yarKytQvqyaVjJ3U9w/t2gPNsRm4eqzTsHip69U1ToYdjRX8bLuauTVvEStZ7j11rR1h/HtzvrRhf69+hA+uvciPHLFGZpsW0tlVXU4pau5MbgaXUnpZ9CMfL/OIdEukCcvcCKzqBKlVTU4+9RuPm8r7GguJv9yAEl55QCAXYl52PbmmBbLTN9wxOf9+Du2yBMBmHDNYLNDID/SoV3bn1YJYEt8Dt5xUoagpbhMfUZlcXcysSE2CwAQlpCHsip1LWiPzmtbXuM8AM+190ZrTOIb/WP5AVXrV1TX4bgXnX6BwBkTXyktX446h8Rf5oVj5Ad/YFNclnYb1oizkx8jPg1JuWW49pNNuPmzrVgfk+nz9h6dF96UxANAan7b78L2I8pP9AMVE3mypG4n8WIRWVdTi7yLvLK9i1Y8b2fBbZ2k/OvXGLz1836nIzi8uDha8XYX7UpBdEqBVzG54tApwfwpMlWT8hWjVNXW4V8rD+KVH/c6LW0pqqjBNZ9swjWfbMKqffX15EzNvbdwVzKu+XgTQg9nO31czcdyRXQ6th/NRWF5DZ4KidQoQu9U1tThv2vjNN+uNzXr/1h+oOlKo6v+Ba5eZ28+2z9EpOBQRrGqdbQcXcsumMhbUPcAT2JvPb8vJt8xzNB9atWJjgJD48HB1eemXbu2DxxIK9Js/6v2HceS3an4ZJ3vB/h7ZoUpGpu5TWmNi0Nz/cRN2n+h/vNbLDKLXfdJKFV5JcAbap7WvG2J+G5nMn6JTnc6J8CXG48gv6waUgJ/+0H5yZc3fCmh0jCIVv+tv0PxhFAKlksvrMCEBbtVBtbWsVzXHa6NNnfrMczanGB2GACAAgMnXcsurlR9hStQMZG3oD7dgs0OwVQSZozswUyelDuRUzj/3DgrrXE2skdrrhJFVznM4vAUTUaDCDua5/M2GpnVApZfZq2ZXX+IODEL5UonI7hk+zBOPAWOLzYqqxGPSlZ3Zc1qZWqtNS+5UcPqz0sPTOTJcqQ0voWcLfKkhqfykQ5OSmuU5LeuPoZWqIdWHIL5oRK1YebH0pfvr15xGz0cpNqXwNtjciAOc8lEnixIBuRZNdmIh9Ka9s46u+qUjGvzXVEfm6unE5tZ7NWsraShVh9MC5wHOk2w6hwSVTXKhixNzS9HjRdDhtbWObDtSI6qKzbOvlM5JVVweDESVUpeOW6bvhV3fbXdcleNlEorKMeMjUd060ivpwwdZt22GibyZDlSp9qaMeed6vIxnjaQGp5a5FPyvbws7OLMwKg8LCGnFDNDj+KYs0mZFAbx8P92aRuUTXlqUXR2YmeNKy/GxFBYXoPRU0Px5aajipb/84ztuO3zrahTmUx/sCYWj38Tgdunb/UmTADAvG3HcMWHG3D/12GqX59JP0YjPqsU+9KK8G8nfSXs4NmFUZj2R7yPWzHmc9X6JCwkLAk5fl7GxkSeLEdtHn/zsL6KlnM3vrMenfPIfzUektR8asxP0dyTUuKhOTvx6brDGDNti8ea2y3x7mceJUpxU+f84W+xTocbdOdYbhlW71c3Y2xIWBIA+DSvwwdrYiElsCelENtUDocYnXJi5uDtDXMmVNbUYeLCKDw4Z6fb18gqYo6rGznGGdWlNT7v8YTPfD4JsTYm8mQ5UkrFifXUB4bj1vM9J/J/urCf28fV/mi89afzVK5B/sTw6c3d7E+r4R5rHRK5pScu/d83O6xlDK1ORT5dd1iT/dqJlUr+XL3t+WXV2Jda2HJZA+JxZnzIieFWW8cbl+FdmUZxpbkzfartVOrMrNCjWBuTiYjEfLy0RN8Ri1zR4mdDz6s3WtbIVyos37IrJvIWcGH/7maHYCkS7mdwbLu85x+Tv40Z6n1ATky84Sy8ess5OKevydMlkik8jSPvbh13vDl2Pf6NwgmV3MgpqcLUAEzM1VLTkc5T0l9RrU9y8fz3zsf3NsOxnDLtN2py+dEXG48gNM75WPVKbYg9sf7eVidddnL79G1+nyTbARN5C7huaB+zQ7AUPUatCe7g/qPuzf5eumkolk682suIyM4Mb5F3k0Bq0UL5z5UxmLP1mPsYrF4bZCNrD2Zgo4/JoCu7juX7tD7fZs8mhPg2Vr2/vMaHs0ow18PvRiP1z9m7JMBKV82MwkTeZFcO6YW/jTm7xX2BXq9dXyNv7Gvg7f4C/K0KWI2XlNV8bpQkwnUWzpadRSaltNVsq96qrnVg4c4k/Lg7VZPtTVy0x+dtzNue2OY+V536LNGJ1sB9rdybjqLyGhR6OYGRmt/1pNwyPDk/Au+sOODVqDZ2N+2PeIQdVddvQAm1x9a4zGJ8ufEIkvN0uApkcYE9hahBenTq4PJgt+SZq9rcZ4UfXTPV18i3vO/VW85x2WHlrD7Kylv0yLmDmMkHJD2+oVN+jXE9vblFfxIe/yYCO49pN5mUVf0QkYJ3f40xNQYtPwJlVbVILSjHef3sV9ZZVet5CMpJS/YCADo6GQZWa899vwexGfWdQS/ufzIevHygptvXKx/QcquPzgtH0sdjNdyiOnUOiXtmhqEiQMt82CJvgJM8lHWQZ67qiwWAywb3wmOjzkDPzh1cri/g/ofL23ycaXxgavw4qvncuDse55dVN42uYSX5ZdVYH5OJypo6pwnF9qO5qocDtCO1SXxmUSXKq9WXPBnxSlbW1OHGqZtx+/Rt+HpLggF7VKe0yv3r9sGaWMXbqvZi3Hm1GpN4APgjNsvtsoF8vNDqhGR/WhHCEnJbbC+vtCpgk3iAiTxZkJT1k380d3In10k6APznnouw9c0b3S6jVcLx07MnrqKwRT4wZRZrO8mIu4NQYm6Zz53rvHX3zB14ZmEU3ly235T921FoXDau/GhjixGAvLX2YEaLRNFbzX/5lkalIbuhBOfj3+PaLqvx2cQ9s3Y47RDp6jt0+QcbtA2A/MoLi/fg0bnh+P1gZtN9/t+U4B4TeQNY9bK4VUnINgfBsRef7nE9T30L3I0aoiYfv+LMXl6tR/6lps6hWQtbezfDNN32+VY8973vNdXeaJzY6td9x3G80P9nSNSCrx0hm5u4aA/+PGM7Moq0mym31ODhG6NTCjFnyzHFLbKB0LLqz+Wzrp6b0mf87sqD2J9W6PG39XkVv4n+/HoDTOQto//JnZpuB/ooNs5aud0lOo08LREIJQBkHLWfJ3cHE3dXdowoD1Dizq+2mx1CQKpzSHy2XrsJbcxofNDiqoKRvH2JtM4Xc0ur4cth60BaESIS8y2RyCoN4dudyRj31Y6AH/RDDSbyBnD1+X32hiFNt0MmXI7R5/bBc6PPwpjzTnW6/Pzxl+HmYc4f8ydTxl3Q5j5fv9MSHlrkOWoNeUGrg007NRMnUMDRajSjsKO5TstpyP/sSSnAnV9tx4NzduKPQ+5r97VkRgJugfMUUzGRN9E//jSs6fbQvt0QMuEK/P3285wmh0snXoUx5/XFvCcv1zWm2y5oO0vq3Ccuw4Lx+u4XALoFt8fKF67BWX26epUge1pHjxb5QByzluo1n3pdCVefvkAcss5fdO7YTtsN6vRRKKuqRVlVLR6d5/vkYd7ip9xYL/1wYsbYZxYaN0mYVq3/PLIqx0TeAGo/162Tw04d2uHywb1cLO3ZiDNOVrTcyEE9Mefxy9rcf8v5fXGji6sEWrr5/L4YPlBZrGpJCZ8uUbqipOSH/NMjc3ch5niR4uUzipzXmI/6aKMms7OS8ezS2f1fK2P0mWVVhSNZpabuk+95FQAAIABJREFU3wqM/Lh4nHHV6EntdGg2X74nTbcZku2E48jbwNKJbcea90fefNGDmp2Kum8dl7q0fAYxkQ9ou5MKfN5GTkmVy4l8yNo8DZVoFXtTC1v8Vrqn/e9krUPikbm7NN+u1valFuL91YcQlezt99r86w4Oh8T7qw9pMmqSN4wsrXn1p32IzyrFk1cPMmyfVsQWeQu6pFkL+sUDeuDC/j183manDp4vAdsxJW2evHssrdFo1Boi8n+fu5iATi8Oh4TUMRF01w+jorpO05FxWksrKPe8kAU88PVOH5J441TVuO4Av2r/cVPnpGjdIFdeXYtvtieiWOVoSUqPyV9vScCa/Rmqtu1v2CJvCHU/zl2D2+Pn567G1vgcPHDZAE32/u1TV+DBOTtVrzv6XONG0HE7YZOL0wylX3Yp3dfIM5Enoua+2HgEd1x0Gs7t182Q/Y2ZthlJefolvPtTnZeBFZXXYPTUUBRW1OCrR0bA3WSolTV1SM0vx9C+xrwmRlM7QtRbP6ubX0Gr6pJpf8S7HKZzY6w5c064MntzAmZsOqp6PTX9z9RMEuaPmMhb1MhBPTFyUE/Nttd87HNXnCWzRvYG96byRellPAn/H0uWiLR1JLvEsERezyQeAN50kXR+sOYQCsprANRPtvP1X0a0eNzhkJiz9RgyiirwS3Q6Sipr8fqt5+DFMUMV79sfhxKcu/UYluxObXGft4eY7OJK9OkWrOp1mrXZ+ay8Sqs99br60/o5eJPEA8DT30VqEU5AYGmNAeycP1o9dDWHB44jT0TU0tKotBb/b328WnMgA5+sjcN3O5NR0lAeMVXluPb+l8YD//lNfStwY47b+jW+4sONeEqjicTMPmlS2mBWVFHj9nGtZ8/2Z5ok8kKI+4UQM4QQ24QQxUIIKYRY5OW2Bggh5gshjgshqoQQSUKI6UII7ZqnA4xdTiSa/wAovazW/DfL3e+XlMCoIac0/b931+CW2/HLQw0R+cIqv53L96QbOhZ4c99qUG/thw3yTrX+uExbf1jxuqGHc5CU69vIQlJK27zWw99bb3YIfkOrFvl3ALwI4BIA6d5uRAhxFoAoABMARAD4HMAxAJMA7BRCnOJmdcuyyLHAI2fJrGXKUVz8OKkZ/m3Yad3x3rgLcMdF/fD9/43SKDAiIv2VmzTMXq0GVzLtklxqTW1Zia/v8e6kAjZKBSCtEvlXAJwDoDuA53zYziwApwJ4SUp5t5TyLSnlGNQn9OcC+I/PkfqB+eMvUzV+eePPsNVnhfXmcKH2J+vJqwdj1mMjDat7JSKys72p6iY+c8Yu4+3bXXFFjfIaeY3b6Fbu9boNl3ykSSIvpQyVUh6RPjTfCiGGALgVQBKAma0efhdAGYDHhRBdvA7UT4w5ry92Tb4J/73/YlXrfXjPRTpFpBGvOrsq3bT7jfM4Q0StWeR6pKH0eM5m121bibu+Wr52QBXC/bEsv6waJZXua9O9NWnJXuSVcj4MM1hp1JoxDX/XSylbjAElpSwRQuxAfaJ/JYCNRgfni2+evAy1DonolAJ8+FucJtvs3TUYwe0Vnoc1nF+d2v0kt4sN7dvV1aqG8O5HTOGoNYF4RCYisoBASeM9tWVmFVdh5Ad/oLBcn2RaCNf9vaKS8/HI3HB0CBL4fdL1Tpepr7H3/t1K9LHGn7xjpVFrzm3466o7/JGGv+d42pAQIsrZPwDnaRGoWpee0ROXD+6FywZ7HgJSDW++cP937ZlO7x/YqxPevL3+5Xn2+iFN97845mzvgvNC899ApU9Nq4lVA+VAQ0TKWaaPkAWpeW0CZQJsJa+IuyTe1/r22IwSlzP4jp+/G9W1DpRV1+GNZfucrFuM6z8NxV0zd9hmxmKqZ6UW+cbpS53PWnHi/pNdPE4uNP9xefmWc1BV68DCXclN9+1++2b06tKxaea/l24ait5dg3Fq92BcOcS4/sXujguuEvvmJzPt3cxBzuMxEZF27pq5w+wQqJVP1x3Go6POcPpYSbPkPCW/HP1P7tTi8adCdiOjqBKp+RX4YkM83h57vq6xknas1CLvSWPG5jElk1KOdPYPgDZ1LV4yq1HiqWtOtMJ3DW6P9++6oMXjnTq2azF9d5fg9nj6+iG465L+hsXYmjevVbsggRmPXIpRZ/bCrMdaTmqi59TnRESe5JZW2aKFX2mI+9Nctbk52aaXsQQaLY5T3uYZGUUnxm2PTvG+g3NmMevkjWalFvnGX4UeLh7v3mq5gOfuCztu+Ono2z0YPbt0xLjhp7dcr1XztlUOLs1/xHp06qBondaXbO8cfjrubPV8lWBfLCJqbdKSvcgtrdZkW5d9sAFjLz4NMx8d4XlhE328NrCnu7c7LUYIat/Ou20UlNfg6y3OZ5wl/VipRb5x5gRXNfCN80Grm1IuQPXo1AFvjz0fz48+G0E2KVDsEnzivPKJqwcrWkfxqDXWOFchIpv59+pDmm1rzf4M5JRYu8UyNb/C7BBsywrHmeOFyt4/d6F2aOddajgzVN24+aQNKyXyoQ1/bxVCtIhLCNENwDUAKgDsMjowrRj5HVdzUq53XNcN7e1xmY7tgvDWn070Re7csV2bZc5zMvY7J78gIjv5ISLF7BAMZ4UEN1BsjMv2eRtq5qlprrLGnEnLAp3hibwQooMQ4ryGWVybSCkTAKwHMBjAC61Wew9AFwDfSSk5vlEDrcpBWg72qb137zwf/3t8pMvH7xx+Ora/dSNO7XZieMzWT00AuPosJycEmuXxyjb0yBUDtdohEQWgz/4IvIvKVs7jlbZgB5L2XrbIc74Ac2hSIy+EuBvA3Q3/7dfw9yohREjD7Vwp5esNt/sDiAWQjPqkvbnnAYQB+FIIcVPDcqMA3Ij6kpq3tYiXWtK7I2jHdu1w6wX9XD5+wendWyTxgPIfBCNnDPz8oeG49XzXz4OIiOzl6e8iseal6zTZlq9H0pAdSVqE4bOCsmqEHc1FUUUNbjxP+YzwNqni9TtadXa9BMCTre4b0vAPqE/aX4cHUsoEIcRlAN4HcDuAOwBkAPgSwHtSynyN4vV7ar5PepeneJNrK/1BULppLS7t3nPpAN83QkQUaCxcWxNzvNjsEJosjUozbF/uBrmITC7Ao/PCAcDlcJbOGNmwRidokshLKacAmKJw2SS4yb+klKkAJmgRl9VY6SN+87BTsSE2GyPOOBk9OisbIcZIbUbWcbmcVvvTZjtERNSSddN48mRxuPI+HVZtkff3z5+Vhp8kA818bAR2JxZg5KCeZofiE6VXEziOPBGROSzcIK+psKO5+GRtHB4bdQYG9Oxsdjgu6fZ+sEXMFFYatYZU8qUkJrh9O1w7tDc6ORkdRmtafLdd/fAobQHw9MPFnx8iosBUU6fNiA+1DonZmxMwcVGUJtuzG6u2yFs0LM0wkfdTduo9rqh1wMUy/Xqc5PwBDfXo1AHLJl6l+36IiPyR1a+IDn37d8zerN1ERgfTrVN370pljfbD1Vm1Rt7anz7fMZEnnz11zZm678PZgeDZ64dgSJ+uCtf3zqBTOiP6n7fgssG9vNwCERFZ3Sdr48wOwVAVOoz5bs003v+xRt7GrHLy+5crz8DRnFIkZJeiqtaB3NKWMxfqdXXgH3cMU7ysux76QH2M3zx5Gf76bSSCBPDTs1fhWE4Zrhna2zYz4xIRESmhx1HNqi3y/o6JvIHO6XtiZtKBvTqZGIm22gUJfPfUFZBS4pmFUfjjUJbm+/C1c46SH5ibhvXF+leuR+eO7TCgZ2e2whMRaSBQOrs2t3BnktkhuCQhdcnkrZrHH84sMTsEXTGRN1CX4Pb46dmrsDE2Cw9foXxsVqtr/JHWsy7f4cWRYNzw0/HrvuMYfEpnXNS/h6J1mp9sERGR7wIxkf/nyhizQzCcVVvk45jIk5auOLMXrjhTm5ZeK35lnP1ge4pTSUcoCfVn+/+9/2KMG346Rg7q6bE8xoqvJRERkdayiqsAVHlcTi2L5vF+j4k8acpZLbpWw0+qbdU5qUM73Hx+X0XL8geIiEgfrftNkX+yaou8v+OoNX7KyO+TEVdNjRjvnoiItJddwkQ+EFg5jw/ZkWh2CLphIm9jVvnSNG+Fd5bUeztx1X/uuRCn9TgJb9x2LroGt7fM8yUiIqKWrDx/zZRVh8wOQTcsrSFNuRvmcdZjI/D893vQpWM7lFV7HsP2sVGD8NioQVqG55Ivs+QSEREFOo7UbA62yNua62+NkYlp89Td3ZCNd1x0Gra9eSN2Tr6p5foBOKIBERGRP2Eebw4m8n6k/8knxqa/bmhvU2L4v+vOxA3n9GlxX/OrbQN7dUb3kzoYHBURERGR/2Ei70dCJlyO0ef2wfOjz8Loc/t4XkEjzVvUg9u3w7dPXdHicTucpVu4tI+IiMjyeHHdHEzk/cjQvt0QMuEKvHn7eaZ3Orn27PorAuf07Yo+3YJNjUVv7955vtkhEBERUQBiZ1fSQNvz8JmPjkDo4WxcffYppp9U6G381YNx8YAeuG/2TrNDISIiMoV/H+mtiy3yNmaV/NhZZ9UenTvg7kv749RuJ3lcv7bO3hfkhBAYOagXXrvlHLNDISIiMkXo4RyzQwhITORJsdmPjXB6vzdp+KSbhjbdvndEf0XrWOS8xaW/3TQUm18fjVsVziZLRERE5AuW1pBit1/YD4ufHoVOHdrhnllhPm3rudFnYdApnTHolC4Y2KuzonX0bLdvp9EAuIN7d0H7dlY/5SAiIiJ/wETexoxOF4UQuPosbYa1PKlDO9w7YoAm2/LWref3xfpDWbiwf3ec1sNzCRARERGRlTCRJ58ZNaGT1icuMx69FOHH8jFyUE+/75BLRERE/oeJPPlMGjR6rNZ7CW7fDtefY9x4+0RERERaYmdXG2MrMpG+unRsZ3YIRERELjGRJ9vgaYt/UTpakZZev1XdEKFnndpVp0iIiIh8x0SefGZUjXwg+PfdF2L1367FqDN7mR2K7sYNP12X7V7h5rV74cazVW2Ln20iIrIyJvI2Fmgt1H+6qF/T7eEDTzYxEv08fuUgXNi/B07p2tHsUHSnV5Ls7nuhthzNqP4fRERE3mAib2Pnn9696XYHE8cuN6rVcuSgXnj/rgvw0GUD8dUjlxqzUz+1bOJVZoeAiwf0ULV8/5M76RSJa2yRJyIiK+OoNTZ2+smdMO2B4dgUl42JN5xlWhxGtlo+cdVgw/ZlJj0TyNduOQeXDTavdOeKM3vh2euH4JSuwYqWj/v37TipQzvsTMjDI3N36RwdERGRfTCRt7n7Rg7AfSPNnViJrZb20rubsgRaL++NuwDDTuvuecEGJ3Uwb+QYfraJiMjKWFpDZEF6JpBBJneu8Pa5mTHaKvN4IiKyMibyRDp76pozVa9zwenKW6zVMnv+AW9LscyIWrJJnoiILIyJPJHOvGkBf/r6IdoH0iDI7ERe59xYq6e3YPzl2myIiIhIJ0zkyWdstNSeN3Xho8/to2g5u5bWGO3G8041OwQiIiK3mMiTz4I78GNkBQ9dNlDRcqa3yHtbWmNC3HY56SAiosDEDIy8Mv7qwfj/9u48TI6qXAP4+3XPvm+ZmcxMZklmksk2yWQmM5nsk42QBQJJCIQsLIGERRAQkIAsilcWlU0UARFB8bKoeC8iooigIuAVxasCKjHqVRYBlS0Ekpz7R1VPenqququ6q7q6qt/f88zTSVV11anTp7u+OnUWAOhpqUQHp7HPCFZjTo/jeF8Fx5wQioiIMhmHn6SkXLJ6Eo7tb0ZbTbHnnSczXbpCQasBstefV7L5YSXZq7pG47W39yZ5hJH8dNNBRETZhzXylBQRQUddKXLCLEJ+E6+N/CcOn+z68ZMdCcbK7celh03GgQNJ7d7Qecs7ndsZERGRwxiFEaVZ2KXeplabgcRrI785DTPnulnJXVOSj30ORvJLJtbiqnVdqCnJc2yfRERETmEgT5Rmd5zQB8C7turJ3kdsd2hITLebq+w74NwBRARH9Y7BiXPdGw6UiIgoWQzkiVwWG7jOaa/BD85egMc+Mohxo4pdO46Z3CSbQ12wYmJS7xsp2VFrrG23bz8bthMRUXZgZ1ciD7S7MNKP1fB1/nhr4827JfkaeWuR/P4ENfJ1Zfl45U17HWI5eg0REWUi1sgTuaw6g9pXHzmj0bRG/pLVk9KSBrdD4g8StJH/6on9WDO9Acsm1bmcEiIiIncxkCdy2Qlz2lBbmg8AuHLtVNeOk2g0mOPntOKyw8xHpYnMDeC21urkmhNZbVqTqEa+o64U1x7djaP7rE2gBQBi8WkAERFROrFpDZHLCvPCePy8Qfzjrb0YU1U0bJ2TY7p3NVWYrivJz8Elq+MPLen2+PLTmsqxdXYrRuk3NXZZTV1bTTH+/Pq7SR2DiIiC6b0P9qMgN+x1MhzHGnmiNCjIDY8I4p1wwzHdWDKxDp84fDLaaoxrusdUFeJr2/odP7ZdN2/pxZEzmuJuU5Rn/iM7ob7U0nEuXzMFZQXO1lEU5vKnkojIz7742C6vk+AKXp2IfKyrqRy3bu01Hf/9xo0z8Pi5g5g2xry2HgBK8zPj4dxTOxebrivKy8F9OwYS7qOpsghP7VziZLKwYWazo/sjIqL0evb//uV1ElzBQJ4o4NxuMuOk0oLcuOt7W6tw7iETEu6nME7NfjKc3h8REaVXov5TfsVAnijAxtVa7FiahljfP7cTREQUNAzkiSjjJBpNpbO+LE0pyTwXOjaBFRER+R0DeSJyXKq11E61mglibfmW2S2Y11HjdTKIiCgDMJAnIsftjKo1ttL2m6zLzwnj2P6WpN47t503AKk6f3knfn3pMow1GU2JiCid9iWYLNCvGMgTeWjhhFG4dsN0XHbYZJw4t82zdFjtEJvsGPDaQZJ/a/Ls18AsmViLq9d3uZCW7HLKwnEoK8jF/afPSfuxn/joorQfk4gy2/5gVshzQigiL4kI1nQ3ep0My0107jyxD1c/9AIeef5VdxPkoVu3zvQ6CY4pzA1jzwf7PU1DWYKRiFw5ZmH6j0lEme2t9z7wOgmuYI08UYB845TZQ/++ceMMy++zWlneWV+GLx3nn0BXBbQGxqpwKIi9HxLLzrMmongOBLSNPGvkiQKkp6USf/jkoXhzzweoLkmuGcztx8/ELT/ehbxwCI++8A/H0hby0Xj2blrUWYuckODh373i+rGCGsevmd6A+3/1d9P1LGpEFCuYYTxr5IkCJzccsh3ER7eRXzihFl/bNgtHzmhyNF1hh6KrnpZKy9vGO+S0pviz3brp6vXT8Jn101w/TiigkXxJgXN1UKcNjnNsX0SUuYL6hJaBPJHD0lnz7Obwk06fhlP5MmtsNU5daC34ivfDXV2Sjy9t7XUkTXYopVBemIu1Pc7eKBlx6uYp0ySaPyHR+minDbanmhwi8gEV0Dp5BvJEDjt/eedQEOz2pERWR5tJat8OtzQWB39tzlve6ch+Fk+sQ0t1kSP7SpeG8gLL2wa1Rt5JRXlsYUpE/sVAnshhY6qK8N+nz8VNm2bguDmtXifH1NqopjMb+5tHrM/UGnk7MrH+JdWbr4fPXmB526DG8U4VJb/dxBFR8oLatIZVEUQumNJYjimN5V4nI66LVk7EAaVQkBvGqQtHNi9wOgbM1KAy3clSKV5N7KQ3qE1rErH6CL2xohAAcNOmGTjla88E9kJPRMEN5FkjT+RjlUXJj5ddWZyHazZMx6eOnIrCvPCI9YGokbfww+2333Y72dhto2NwkOSE7F3alk8ZjcfPHXQpNURE7mEgT+Qzd5zQh4UTRuH6Y7pdbt9rL/C+7bheHDatwXS904G8Fx1V/aS9tsTT2YLdlKgk5eVYu7Q16DXygNYkjojIbxjIE/nM/PGjcPvxfXGDZi8s6qzD9cd0m643alqTSmy/eGJdwmA+iC1LrHZCvuOEPuRbDGi9tqF3zLA+G+lQUZSLCw51ptN0NssLh3DRSnc69Y+tKXZlv5SdUm3WmKn88StPRGmXjqY1qbbhLiuM37QoXb/bo22MJJMqO1nml0m4Sgpy8On1XXjs3IU4ZHJdwu1T7TA8tbEcT16wOOlJ0yiKAFtnt2L2uGqvU0IUVzDDeAbyRBlrTru3F8Z4odLnNprXvJvuz2CHmwdahv59rMHIOalya1SSZM4/3UTMA/l0TEZlh0ALzluqix0f9tRIdUkeCnJH9gvJJOvSMM+AE0KiTUJ3K5u6UYYLaIU8A3miTHXNhum4cMVElOZ7M7hUvFrPlVNHY9Mse4G30f7OXjoe63qacER3o2Njw0eb0liOE+e2oaW6CLdscS7Q6KgtTfq96bqWCAQFuSN/4ovzwqgrS98TBCvs5klXUzlu2tST9PH88JzCzgzGXorcLKbjBixWKp39KftwQigiSqva0gKcNH8sJjaUeXL8eJdlEcHM1irDdYdP19ru51gYb7K0IBefXj8N12yYjvIEzWSS9bFVk/DYuYNYOilxkw03fPm4mabrFnXW2t6f1VYlIkBLtf02xss8yic71kxvxPIp9V4nw1V+qT2MFEc3WnHFy4L+tiqsmDra+YNSYPnlO2WXY4G8iDSJyG0i8ncR2Ssiu0XkWhGxVa0gIitF5GER+T8R2SMiu0TkXhEZcCqtRL7i0Y9PshfmTx05FTdunIFvnjrb2QQZcCJ2cOvH/ep1XfjRRxZiME6w/qFFI8fvN3LLll4sn1yPu07qN6z5rCjKxahSa+29FeLXTN3s4JOLZFgpd5yxNnO42Q8jUefEMMsB2RDQON6ZCaFEZByAJwDUAvg2gOcB9AE4E8ByEZmjlHrdwn6uBHAegNcB3A/gNQDtAA4HsFZEtiilvupEmokovmSvz0V5OVjZNRov/XuPswkyMH1MBerLCvDym+9h+eTMqqFd3zsm4TYlFptNLZ1UN/RE4f19B0asf/KCxcgJCdov/O7Qsg/2j9yO/MM3zQBcjKUT5QADeSLnZnb9PLQg/gyl1A2RhSLyWQBnAfgkgB3xdiAi9QA+AuAVAF1KqVej1g0C+CGAjwNgIE/ZxafXqnQ8xswJh3DP9gE8uet1HOJxIH/+8k5c+dDzQ/92i9ENllHHzb0GAT+gfS6Z9og5+pTSMdBOqqPe0EFuNq2JRyF7Zy6m5GTa755TUm5aIyJjASwDsBvAjTGrLwHwDoDNIpKosWaLnp6nooN4AFBKPQrgLQCjUk0vEVmTKNhpjJpMx8iBNP1qNlcX4aiZY1Cexo5vRllz/JxWnL10PM5eOh7Hz2l179gWt6sqyjNdl2nXs3Snx6zJhpNj2V++ZgqO6XN+JKZME2nm5EVnV7s18nPba/DFzcl3kia/y7RfPmc40UZ+kf76sFJqWBWQUuotAD8FUARgVoL9/AHA+wD6RKQmeoWIzAdQCuAHVhIkIr8w+gPA2T+ILEp0iextrcJh0xpQWpCDazaMHM4wqLUfZgpywzhjcQfOWNxheWhDp7Po5s09mNZUjktWT0JlsXkg77ZUOsx6ERBGrOl2bpK1sTXF+NiqiYbfDSv88v1xtbNrgjyw21eiKC/s+ZO7WOmcgyLb+eU7ZZcTTWsm6K+/N1n/B2g19uMBPGK2E6XUGyJyPoDPAvidiNwPra38OACHAfg+gO0OpJeIHHL9Md3Yf0AZ1ow1VBSipiQPr739Prqayj1InTWrp43GjY++CABYOMH9h35WLiZja4qx67V3AGBE3sV7UrJscj2WJQhUju1vdnWGw6vWdeGQSfaCJTdC98qiXPzz3Q+Mj2eSh0431SjKy8ER3U046+5nHd1vrJyQYN8Bb6KUg8NPpp+VkbGGbR9mUxwKHicC+chV5t8m6yPLKxLtSCl1rYjsBnAbgJOiVv0RwO2xTW7i7Mfw2ZleKz/Dyj6Isp3VdsRmj7fDIcHd2wfw6POvZvQwcacPduCvb+zB23v34VNHTvU6OQCAyuI8fG3NFPz0j69hY8xEWbG5nZdj78Hqh5eOx893v5FiCs0dZaGTbyw3QtDbjpuJo29+0rSvgNvpSGdYHfYwkI/8TKS934GyN2KOiLt9Vyh5126Yjg/f/SvXjxPQCvm0jCMf+aYlzEMROQ/AfQBuh1YTXwygB8AuAF8TkatcSiMRxXDisjxuVAm2zRuLhgTt6b1UmBfG9cd047bjZqZ9oqR4FeNz2mtw3vJONFXGn53Wzue0Y8E4yyPleGVeR03ijXR3bes3XdfdXImndy7Bzy5YlLA/RyayE3R4OXpLdbE27KkbKUg0cs9cG2Xlvh0DSc2rQJq8cAilBTmY2uj801WnR2i6ZsM0PHLOgpHHCWjbGicC+UiNu9mnWxaznSERWQjgSgD/pZQ6Wym1Syn1rlLqGQBHAPgbgHP0zrVE5DIOCOGc6EByYFx1wu3jXXBiPxc7n1PIcrWKd9b3jsFRvdY6nc5ujx/IlRflYnR5IR47d6EDKYvvkMneTaTlZSB/7dHTPTu22aR0RnparG9LI508fyyevXgZzl46Pq3H/coJfbbfkxMKYdyokhHLM/hnLyVOBPIv6K9mn26H/mrWhj5ilf76aOwKpdS7AJ6Glt5uuwkk8jOvLtFedjoMmtuOm4kpjWVYOqkOJ8xpS2lfqTRhiLw1k8coD4cEV62bhioHO+vmhIdf6two2Tdtcng0FBu1h3bbijtp4mitri79LWsytwwHUUi0zsXpzvdkRj8za3IV0Ap5RwL5SOC9TESG7U9ESgHMAbAHwJMJ9hOZltCst1lk+fvJJJKI7GGNvDm7WTOhvhQPfGgebtnSa7tNu5OCeHN22uA42+9x43ru5dj04ZB3ZSrCrfNfMtF8ZmSrxo5ik5qUuTqDb7yV9vdn9oSKTWtMKKVeBPAwgFYAp8WsvgxaO/c7lFLvAICI5IpIpz4bbLQf668ni0hj9AoRORTaDcF70GaQJSKXBS/k8w+3LjdDNfIBup6ds3RC4o08kM65ecuZAAAgAElEQVQ89rJG3k1dTRW4ep3x8J1W8ndV12isndGEW7f0Opyy7ONVCUvmCUC2zfjrVK+nU6EF2NeLyGIAzwHoBzAIrUnNhVHbNurr/wwt+I+4D9o48UsAPCci3wLwMoCJ0JrdCICPKqVedyjNRESecuIxtZ0hEyO1pkEK5EMhwayxVXhyl/WReMxyzKt8aawoREt1EZ548eDlzc3OruWFucgJCV5/J7MfcP/HmqkoLUg+TDmmrxlzEvSjSNWzlyzDtMsedvUYmcDOCEF2xdt1Mt9Js69DgH72hnHkeZxeK98LbbSZfgDnQBt15noAA1aCb30yqRUAzgLwO2gdXM+BNpHUgwAOUUpd50R6iciC7KrUyAp++kirbbSRv3zNwWFDr17X5UZyXHHl2qnYfcVK/Pi8QSyZmLiz7O4rVuLyNVNGLDcL5NtqjJuUdNaX4ucXLrGXWA+UF+WaBnlBDcoylZstx5y+gTadKCyghcaxhnVKqb8qpY5XSo1WSuUppVqUUmcqpd6I2W63UkqUUq0G+/hAKXWtUmqWUqpMKZWjlKpVSq1SSgX/lpcogwSxPbVbKoqcnUXVrZrhg51dM991R3eb1qzFaq8twfc+PB93bevHuh5rI9444ZNHTEFPSyVuSbLpRnlhLgAt8Ng0qwXttSXIDQuuP6bbVhnYOrt1xLIzFrXHHXbT7qyo6dZcpQ276mXfA0tS/DL55elYyKNmeckcz+wp5Vt796WYmszkfQ8ZIspIU6NmFO2oHTmUV7a7Qx8WLSTauMV+EBpqWuNt9GBlrPhJDWX48fmLLO9zQn0pZrfXpC3wEwGO7W/BN06ZjaWTUh96Mi8nhIc/PB//c+FSHDatwdZ7V3eNnHBNwbzplk9ix7hSKcN9NoatjBU7/GJJQY7lG85027HAfkdwM141y0vmcG42A8pEDOSJyFBJfg7u2T6AUxeOw5e2zvQ6ORln/vhR+MHZC/D4eYPorC9L/IYMELm81aZ54qtom2e14M4TzSdyiubWRE5OXOfXJ6j5T6b/QygkKC/SaunN0hjbsfW6o6ej0uZQnV7eyH3yiClxJ/JyktnHnEq7+9gmUOGQ4L9On4ttcxMPK2t0A+FmzDmztXLoyUYmK8wNm65TSqFC/05YlQGDOKVVlp0uEdnR11aF85Z3ork68y8G6RS5+LbXliSceTWedMdTkXRPH1OBVV2j415A3U5DJkoUfJ8+2I6lk+pwyOQ67FwxMfXjxTlc7LpIn4HYMfEPn97om0ZwK6bWY2Nfc8KJvPxmSmM5zlvemXC7e3YMpCE1BykF7D/gzI+Mm7XcpQW5pvNGKABfPs5eRZKdAQCCgIE8UYbLst+kjBWZmrwkP8f1qd7dG37yYGH63MYZ+PWlywxrCe10NLWdBtf27L6KolzcsqUXX9zc63i/iEQ2D7QAAMIGV22j5kQT6ktNbxS8qpDfuWJi5rd5T1Ki09rY35yehERRACY3OPO00M2afQWFc5YZzymqlFaW7cj0/h9OYyBPlOGqS/ITb0Su++LmHlxwaCe+eeps5BpFUz4QG2zkhkOGkfUtW3sztt1vJquJ+q5ObjjYx+SI7kajzeMya/5iZfKnma2VWDl1ZLv5oX3bTk3maTBodnXOUrMJ5odL9vxrSpK7efvCsTNw8apJxmlx+cO4eLXxce06dEo9AHfKTqI8sDvwAtvIE1FGuXjVJBTkal/VmzbN8Dg12auhohDbF4zD+Dp7tUNei55Jtqe50tJ7ZjRX2upomkhnVI3aEgc6hmaqr5/Uj439zbh1S++wpgJmEzYlExQZ7St2yY3Hzohb8+1UG/nBCWYTsRtzqja+tCDHMEDNz40JaRw43KFT6pETEpQV5OD+0+aY1rzHO9ShU0ejwINmbEopjCpNvSJobnuNq7XcB5SKE6wrW0+lQ+LcUwi/cGpCKCJySV1ZAZ68YDH+9e4HaDUZF5oCxsFqunu3D+CTDz6Hma2V6B9bPWK92fXZyY6mN23qwacffgGd9aWY12Ev+POTjrpS/McRUxNvmAIrs7iG0jDCyNoZTabNIZwQr7/CUzsXoyhvZPgyIhh04Pzrywvwx/9YMfT/5156M6X9iQz/XOwEqRv7m3HXU3+xvP0B5cwwwk5MXBd//+b5YLeJ/3+dPteTmyYvsUaeyAcqivIYxAdQOtoqTxtTgXu2D+DcQ4w749m50F+1tgs9LZUozc/BzZt78PBZ83HSvMSjdbTWFONzG2fg9EUdlo/lhUwa09ssKTlhgxr5mEWJOvs5cZqfOWqaYfMWq247Lrmx91d1jTYM4t0S+/1I9aHCtRumD/3bbNjaTxw+2XD58sn1qR08U9no9B3PDcd0Y0pjeeINA4Y18kRElNChU+qxrqcJR80cg/f3HRjWZMdNx81uxe1P7AagdTa9aVOPA3v1bxva2tLEQ4cmaiPs1Q1LdKoWddbhgQ/NxaobfmJrH5esNg5yAYMg24WP2SzvrDYbWtXVgANK4cABYHVXA6566IUR22weaMV///olPP2nYfNpGt5EnLVkPK75we/NUuuLwRIUlOlHNb6uJOE53LWtH+++vx+DnbWOp80PGMgTEWWYdMZZVi/0s6PayUYH8W4HhZesnoRj+5vRVlOMAwppu4EAgG+dOhtHfP6JlPdjlkV2hp+MmNJYjpVdo/H9376Ci1ZpQ2DGBpESihw3gx4xGEim9jTeOUV3MI67DxuF1umm4eGQ4Iju5GYfNmpmcvL8saaBvBvfTTfmIFDK/Heoo64Ue/ftj/v+oA1nahcDeSKiLGY1kDfviuYuEUFHmjoYx55Lt8XOwW6YP948OLlx4wy898F+07bACWvkU0qZx2ISf8/2AXz8gd+ir7UaA+NG9gFJVWxWRg/DmGvQzMmx4xosiw2iz146HoV58duDO5HC6OZFbpSd3pYqfO+3L1s6Po3EQJ6IyCMddSUoLcjBW+/ts/W+ma2V+PnufwIA5rhYG7VkYi1+8NyrAIDlU4zb5zpRQZdJbdMzRXtt/JuXeB36EtYie5ThTjTziB09pa+tCg98aF7qO7aoOD8Htx8/Ew/95mVsmtXi+P7P1yeWMsqr2I/tjMXx+5xonUgzOwheMbVem83YIJndzRUAOJdKIuzsSkTkkdxwCN88ZTYuWjl8ltBEcdZnj5qO9toSdNaX4vI1U1JKQ7za2yvWduG85RNw98mzho2RHu0Ao/CkudX0JR2j1jilxWTW6Oi0r57WAEAL2s3KoZFka3Kjhw41araxcEItrljbNaxpUNihNjjre82b3dgtL159/o0VhTh+TivW9yRuQhSZkM4o964/utvhlAUTa+SJiDzUUVeKjrpSXP6d5yy/Z0xVEb5/1nwA7ta41ZTk49SF7a7tPyLdNW6ZUsPnVqCV6Py8iu+NAusvbe3Fiut+gvf3HzB937UbpuPEuW2YNDo944PfcUIfrnzoeUxuKMfC8ekdLjUy2VxsXp2xqD2p8uJI0xqbO7lnx8DQ8LVbBlpx5UPP4yd/fM32ccfozZhiDx/9pJBYI09EGSY/qjOjm21Q/U5EMuKxuROd3/xQcww4V+vqtoTDT7qU33ee2IcF40dh29w2rJ3RhFu3JB5isr22FF/d1h93m3BIMH1MhWMdnROd/oT6Utx5Yj8+eminZ9+x2MOevWwC9tscVF05NGpNcdRwn1bKTvQcFFObyhN+vkD8ConYdZk+jG26MZAnooxy69aDF/8vbZ3pYUq8k87RRlINVI7uax7698qu0akmx1NGNyUnzx8LACjNz8FRvWPSnSTLts1tQ0i04Tpz9Fpd09FyXCpf8zpG4Ssn9OGiVZPwmaOmWZ7F1yjfvbxH9fJ2Ld55223G5tQNW0nBwUC+rNCdhhx28twn99Npw6Y1RJRR5rbX4J7tA1BKoa+tyuvkBF6q18SJo8tw48YZeOGVt7B1ILnOfxnwYMHUR5ZNQG9LJSY1lKE4P7lLZjqeOFy0ahLOWjp+WBpbqopGjEWervQYMfucjZITPXFSNjLKqzitjww51dn1uNmtQ/8eGFuNaWMq8Oxf/5XyfuOpLzs4X8LI6QEy+AfDA6yRJ6KMIiLoa6tC/9jqjGg6EnROZPHKrtE4e+l4VNvoiBgtk5vW5OWEsGxyPZoqjTtlZpLYG40LVkxETUneiCYpXuW31ePeta0fMzwc+jOdvzvb5o01ToNBsLrfgw/uxo0zhnXqFRF865TZ+OxRxrPS2lWl/2bEZvkdJ/ZFHXP4Ol4WhmMgT0SUYdJ5veY10Ttuf8xVxXn46UcX4emdi5N6/z3bB9ISNDWUFw77/+z2GkeCaT8EfJtmNQ/7f7wkH7DbRj7FH5Lt88caNpcLhWRYjXmyJo4uw8qp2v5jP6vxaZo7IggYyBMRZTE+9Qi2/JwwKoryhi2zGt71tVXhm6fMdr2ZS3N1EU4fbEdLdRFu3DjD1WMBiW+U0/mNyAsbh2HGTWsy59GV3ZREd4CN+M6H5g51II/XXEZE8PWTZuHI7kZ8/aRZNo8cfAzkiYiy2OKJtUP/7qzPjlowr2apHXE8j9q42Dlud3Ml1nQ3upgazUcOmYDHzh30fYdpN3nRtMZMnc0a+TujmspExE7uFc/AuGp8dsN0V2bv9TsG8kREGSad1+ujZzbjmL4xmNtegy9s6knfgaOk46FA9LCmXU3lcbZ0XjpHIaLUpfMhVewTscj/N0fNGnuYPiGW/aY1ibeZ3JDc2PzttSXYMtCC6uI8LOqsTbj92FElWBpnFCM+GEweR60hIsowoTRWsYRDgk8d2ZW+AxpIx43Lf548C6ff9Us0VhbipPnGHQyJMsXSSXW4aOVE/P1f7+G0wXEA7NfIW7mBTCWA/vjhU3DZYZPxwK9fwg+fTzxBU7zkd4852Lm5OC+cfKKyEGvkiYgywFXrDgbTV671NrAOou7mSvzk/EHcs30A+TnZHShEAqqr140sZ4W5YYRDgtuOSzyZk1/54fmIiGDbvLG4ePWkodGg7NbIW6EUcOnqSUm/304fm9Zq85GfmquLcOXaqVg5dTTu2TGQdHqyEWvkiYgywNoZTagqykNlcR4mN6S36Ue28Kpj77SmCnzzmb95cmwjkZra9b1jMGtsNeZd9ejQuqcuXIw97++P2wY6aM0gvnJCH7be9vTQ/zO1A/jo8pEdRuOxWoG/sb8FOeEQ8nNCOPe+Xx9c4XA2nLmkA9/9zct4/Z29+OLmkTeKG2Y2Y8PMZoN3UjyskSciygDhkGDJpDr0tHg3fja549j+ZgxOGIWWODWSXhlVOnzs/5K8HNsdGTOV1Th0fkeNq+lwyuKJtZg/fhSK8sK4/pjuoeX/efIs5OeEUFUcMzqRxUA+LyeETbNasN7lmYtLC3Lx2LkL8fMLl2DB+FFJ7yeD+vxmBNbIExERuSgnHMKXj++DUgptFzzodXLiBkJWKqMzs746eSKCBz40F9/+1d9w+HT3R+hJlojgjhP68P6+A8Mm+Zo1thpP71yC/NwQOj/2kIcpTCwnHEKpyZCblBwG8kRERIDrjaczpclGvNO0ksZMOQ8nTWksHzaDaSaLnakXAMqLckcss1KcWbvtf7wtIiIi8oCdIKqtptjB49qP3i6J6hB52WGTHUuLF7wavz/dUj3PeJM0UeZgjTwREZEHGivjd1780tZenHPvs+hqqsDqrgbHjhsd3lnvENmM/QcUwiHBhpnutqV2ShCfHDjNj1mU7LwMgxOSb5efyRjIExERpcnnNnbjY/f/BgvGj8LM1qq42y6eWIdfXLR0aBp7x0TFQVYDufycMLbN4/j78Zy5uAPXPfIHr5MxhE1rhrsioMP6MpAnIiJKk1VdDVgxZbTl6ekdD+JjFOSGsXpaA/772b9jg8ujlqTDIZPr8L3fvoJRpfnobq5I67F3LBiHkAjyc0O4/5d/w/Mvv5XS6Cwpy6IgPZFxo4oDMxpTLAbyRERESP6RvV1Wg3i3xJ7l9UdPx0cP7URjhb1xyjPRlWu7sGD8yxgYV43cNI+OUpgXxplLOgAAx/Q148ldr2NeBgxtefycVnz5p7ttv8+PzW6yEQN5IiKiLBLbCVJEAhHEA0BFUR429ns/qVB5YS4OmVzvaRoiN6aXrJ6M3HAINz++a8Q2mTi3QSLZ1BzICo5aQ0REFHAdtSVD/+5ri982n4IhOuCNnSwqPyeEyqLcuCMQtfowyDcT5I7PDOSJiIgC7gubetBWU4ypjeXYuWKi18mhNIutxX565xL87ILFqI1pN37rll6U5udgZmsl1vX4v89ENmDTGiIiIgDjRpUk3sin2mtL8MNzFgS6ZtKqbGmaMWyY0ZieEUYTSAHAkkl1+MXHlhpOOkWZiZ8UERF5qqQgM+qUWqqLceGKiehtqcRd2/q9To7jGMRnl+gbFjs3L5kexJcVGt+EZKvM/rSIiCiQIjOFlhXk4KR5bR6n5qCT5o/FfafMxux270cboezj1q1WkGazbaspxqqu0QiHBOcv7/Q6OZ7LjGoQIiLKKsfPaUNfWxWaKopQWsAaNspeK6bW48H/fRnzOmpQnO9cWJau4VS98LmNM/D23n0osZhfQX4WxUCeiIg8Mbmh3OskEHnuhmNmYMeCf2PS6DJH95ts0xq/sBrEBx1zgYiIiLJGptVUh0OCribnZ6GNPsuWmmLH90+ZgW3kiYiIiAJgfU8TACA3LDhsWsPQ8lVTR2NeRw3KC3Nxy5Zer5JHLmCNPBEREVEAfGz1JExvrsC0pgqUR43uEgoJ7jyxH/v2H0BOmHW4QcJAnoiIiCgAygpycWx/i+n6bA3igzzyanZ+okREREREPsdAnoiIiLJGEEdwoezFQJ6IiIiIyIcYyBMRERER+RADeSIiIiIKLAnw3K4M5ImIiChrHDp19NC/e1oqPUwJUeo4/CQRERG5rqIoN/FGaXDMzDF44eU38fK/38Mlqyd7nRyilDCQJyIiIlfcu2MAd/zsz1gzvQEFuWGvkwNAG0v98jVTvU4GpVGQx5FnIE9ERESumNlahZmtVV4ngyiw2EaeiIiIiMiHGMgTEREREfkQA3kiIiIiSkp1SZ7XSchqDOSJiIiIKCkDY6sxt70GOSHBJw7PzFGAMmXEJDewsysRERERJUVEcOeJfXh77z6UFmROwHzz5h6cfOcvEA4Jrjiyy+vkuIaBPBERERElTUQyKogHgGWT6/H9s+ajtCAX9eUFXifHNQzkiYiIiChwOupKvU6C69hGnoiIiIjIhxjIExERERH5EAN5IiIiIiIfYiBPRERERORDDOSJiIiIiHyIgTwRERERkQ85FsiLSJOI3CYifxeRvSKyW0SuFZHKJPY1T0S+ISIv6ft6SUQeFpEVTqWXiIiIiMjPHBlHXkTGAXgCQC2AbwN4HkAfgDMBLBeROUqp1y3u6yIAnwDwGoAHALwEoAZAN4CFAB50Is1ERERERH7m1IRQn4cWxJ+hlLohslBEPgvgLACfBLAj0U5EZD20IP4HAI5USr0Vsz6zpg0jIiIiIvJIyk1rRGQsgGUAdgO4MWb1JQDeAbBZRIoT7CcE4EoA7wLYGBvEA4BS6oNU00tEREREFAROtJFfpL8+rJQ6EL1CD8Z/CqAIwKwE+5kNoA1a05l/ishKETlfRM4UkQEH0klEREREFBhONK2ZoL/+3mT9H6DV2I8H8Eic/czUX18B8AyAqdErReRxAOuUUv9IlCAR+YXJqs5E7yUiIiIi8gMnauTL9dd/m6yPLK9IsJ9a/XUHgEIASwCUApgC4HsA5gO4N/lkEhEREREFh1OdXeMR/VUl2C4ctf06pdSz+v9/KyJHQKvxXyAiA0qpn8XbkVKqxzAhWk39DGvJJiIiIiLKXE7UyEdq3MtN1pfFbGfmn/rrrqggHgCglNoDrVYe0Ia1JCIiIiLKak4E8i/or+NN1nfor2Zt6GP38y+T9ZFAv9BiuoiIiIiIAsuJpjWP6q/LRCQUPXKNiJQCmANgD4AnE+zncQD7AHSISJ5S6v2Y9VP0190ppLX1ueeeQ0+PYcsbIiIiIiJHPPfccwDQ6uYxUg7klVIvisjD0EamOQ3ADVGrLwNQDOCLSql3gKFJncYB+EAp9WLUfl4TkbsBHAvgYgAXRdaJyFIAh0BrnvNQCsl9c8+ePXjmmWd2p7CPZEVGzHneg2P7GfMtOcw3+5hnyWG+JYf5lhzmW3KYb8lJNd9aAbzpTFKMiVKJ+qBa2InIOABPQBt55tsAngPQD2AQWpOa2Uqp1/VtWwH8CcCflVKtMfuphTbufDuAHwN4GkALgCOgdZbdqJTy5cg1kSExzTrikjHmW3KYb/Yxz5LDfEsO8y05zLfkMN+S44d8c6KNPPSa9V4At0ML4M+BVut+PYCBSBBvYT+v6u+/BsAYAGdAm3DqOwDm+TWIJyIiIiJymmPDTyql/grgeAvb7cbBISmN1r8B4Gz9j4iIiIiIDDhSI09EREREROnFQJ6IiIiIyIcYyBMRERER+ZAjo9YQEREREVF6sUaeiIiIiMiHGMgTEREREfkQA3kiIiIiIh9iIE9ERERE5EMM5ImIiIiIfIiBPBERERGRDzGQJyIiIiLyIQbyLhORJhG5TUT+LiJ7RWS3iFwrIpVepy0d9PNVJn8vm7xntog8KCJviMi7IvJrEfmwiITjHGeriDwtIm+LyL9F5Ecissq9M0udiKwTkRtE5Mci8qaeJ19N8B7X80ZECkXkMhF5QUTeE5FXReQeEZmYyvk6xU6+iUhrnPKnROQ/4xwnMPkmItUisk1EviUifxSRPfo5/UREThQRw2tBtpc3u/nG8naQiFwpIo+IyF/1fHtDRH4pIpeISLXJe7K6vAH28o3lLT4R2RyVF9tMtvF9meOEUC4SkXEAngBQC+DbAJ4H0AdgEMALAOYopV73LoXuE5HdACoAXGuw+m2l1Kdjtj8cwDcAvAfgbgBvAFgNYAKA+5RS6w2O8WkA5wD4PwD3AcgDcDSAKgAfUkp9zqnzcZKI/ArANABvQ0t7J4CvKaU2mWzvet6ISD6ARwDMAfA/AH4IYAyA9QDeB7BIKfVUSieeIjv5JiKtAP4E4FkA9xvs7jdKqfsM3heofBORHQC+AOAlAI8C+AuAOgBHAiiHVq7Wq6gLAsub/XxjeTtIRN4H8AyA3wF4FUAxgFkAegH8HcAspdRfo7bP+vIG2Ms3ljdzIjIGwP8CCAMoAXCSUurWmG2CUeaUUvxz6Q/A9wAo/cONXv5ZfflNXqcxDXmwG8Bui9uWQfvh2gugN2p5AbQbIgXg6Jj3zNaX/xFAZdTyVgCvQ/uCtnqdDybnOwigA4AAWKifx1e9zBsAF+jvuRdAKGr54fry30Yv90G+terrb7ex/8DlG4BF0C5QoZjl9dCCUwVgLctbyvnG8hZVVkyWf1JP4+dZ3lLON5Y343MUAD8A8CKAq/U0bovZJjBlzvMMD+ofgLH6h/Sn2A8JQCm02sR3ABR7nVaX82E3rAfyJ+h59hWDdYv0dY/FLL9DX368wXs+rq+7zOt8sHDuCxE/IHU9b/Qfvz/ry9sM3vO4vm7Q6/yykW+tsH+hC3y+xaRvp56+G1jeUs43lrfE5ztNT9/3Wd5SzjeWN+NzPBPAAQDzAVwK40A+MGWObeTds0h/fVgpdSB6hVLqLQA/BVAE7ZFZ0OWLyCYR2SkiZ4rIoEn7s0iePWSw7nEA7wKYrT+qsvKe78Zs42fpyJtxAJoB/F4p9SeL7/GLBhHZrpfB7SLSFWfbbMu3D/TXfVHLWN4SM8q3CJY3c6v1119HLWN5S8wo3yJY3nR6u/MrAFynlHo8zqaBKXM5qbyZ4pqgv/7eZP0fACwDMB5a+6kgqwdwZ8yyP4nI8Uqpx6KWmeaZUmqfiPwJwGRoTzueE5FiAI3Q2tq/ZHDcP+iv41NKfWZIR95YKbOx7/GLpfrfEBH5EYCtSqm/RC3LqnwTkRwAW/T/Rl+cWN7iiJNvESxvOhH5CLQ2yuXQ2nnPhRaMXhG1GctbDIv5FsHyhqHv5Z3Qmr3tTLB5YMocA3n3lOuv/zZZH1lekYa0eOnLAH4MrR3YW9C+FKcDOBnAd0VkQCn1rL6t3TzLpjxOR94EMT/fBfAJaB3BdunLuqA9bh0E8IiITFdKvaOvy7Z8uwLAFAAPKqW+F7Wc5S0+s3xjeRvpI9A6CEc8BOA4pdQ/opaxvI1kJd9Y3oa7GEA3gLlKqT0Jtg1MmWPTGu+I/qo8TYXLlFKXKaV+qJR6RSn1rlLqN0qpHdA6/BZC+8GxKtk8C3Qe69KRN74rs0qpV5VSFyulnlFK/Uv/exza07CnALQDMByWLNGubWybkfkmImdAG33heQCb7b5df8268hYv31jeRlJK1SulBNqT2SOhVeb8UkRm2NhN1pU3K/nG8haVCJE+aLXwn1FK/cyJXeqvGV/mGMi7J3KnVW6yvixmu2xzk/46P2qZ3TxLtH2iu2E/SUfeZE2ZVUrtAxAZisxOGQxEvonIaQCugzbE3aBS6o2YTVjeDFjIN0PZXt4AQK/M+Ra0ILMaWsfBCJY3Ewnyzew9WVXeoprU/B7Axyy+LTBljoG8e17QX83aPnXor2Ztp4LuVf21OGqZaZ7pX9Q2aB3LdgGA/rjwbwBKRGS0wTGClMfpyJtsK7ORR9RDZTAb8k1EPgzgcwB+Ay0YNZqYjeUthsV8iycry1sspdSfod0ITRaRGn0xy1sCJvkWTzaVtxJo6ZsI4D2JmhQLwCX6NrfoyyJz2gSmzDGQd8+j+usyGTn7Xym0yQH2AHgy3QnLEAP6666oZT/UX5cbbD8f2ig/Tyil9lp8z6Ex2/hZOvLmRWidhMaLSJvF9/hZZMSoXTHLA5tvInI+gGsA/ApaMPqqyaYsb1Fs5Fs8WVfe4mjQX/frryxv1sTmWzzZVN72AviSyd8v9W1+ov8/0uwmOGUulbEr+ZdwLNOsnhAKWokPX6oAAAKTSURBVI/vKoPlLdB6aysAO6OWl0GrRciKCaFizmMh4o+Hnpa8gf8m/kiUb/0A8gyWL9LPXwGYnQ35Bu2Rs4I2u+CI7yXLmyP5xvKmpaMTQL3B8hAOTmz0U5a3lPON5S1xnl4K43HkA1PmPM/kIP9BG0P0Ff3Duh/Ap6DdeSloj1yqvU6jy+d/qV6wvwvg8wCuhDal8R49D74T+yMEYA20x1lvQ2vjdxW0TmWRL4IYHOcz+vq/Qqs1uxHAa/qy073Ohzj5swbA7frfQ3p6X4xa9ul05w2AfGhzHCgAP4c2Osdd0MbMfgdAv5/yDcCPoP1Y36uf/zXQhntV+t9FJscIVL4B2KqnbZ9+Ppca/B3H8pZavrG8DaXvw3paHgFwM7Rr323QvqcKwEsAJrG8pZZvLG+W8vRSGATyQSpznmdy0P8AjIE2BONLAN6HNsvXdUhQsxOEPwALAHxd/2L8Sy+4/wDwfWhjMI/4kujvmwPgQQD/hBb0/y+AswCE4xxrq/4leQfaMJePAVjldR4kyJ/ID4zZ324v8gbaaEKXQXtqsjfqQjEp1XNOd74BOBHAA9BmGH5bP5+/ALgbwLwExwlMvlnIMwXgRyxvqeUby9tQ2qZAC3B+BS3I2QetQ9/P9Tw1vP6xvNnLN5Y3S3ka+Q6PCOSDUuZEPwgREREREfkIO7sSEREREfkQA3kiIiIiIh9iIE9ERERE5EMM5ImIiIiIfIiBPBERERGRDzGQJyIiIiLyIQbyREREREQ+xECeiIiIiMiHGMgTEREREfkQA3kiIiIiIh9iIE9ERERE5EMM5ImIiIiIfIiBPBERERGRDzGQJyIiIiLyIQbyREREREQ+xECeiIiIiMiHGMgTEREREfnQ/wMCxIY8hymkeQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 248.0,
       "width": 377.0
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses['test'], label='Test loss')\n",
    "plt.legend()\n",
    "_ = plt.ylim()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获取 Tensors\n",
    "使用函数 [`get_tensor_by_name()`](https://www.tensorflow.org/api_docs/python/tf/Graph#get_tensor_by_name)从 `loaded_graph` 中获取tensors，后面的推荐功能要用到。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tensors(loaded_graph):\n",
    "\n",
    "    uid = loaded_graph.get_tensor_by_name(\"uid:0\")\n",
    "    user_gender = loaded_graph.get_tensor_by_name(\"user_gender:0\")\n",
    "    user_age = loaded_graph.get_tensor_by_name(\"user_age:0\")\n",
    "    user_job = loaded_graph.get_tensor_by_name(\"user_job:0\")\n",
    "    movie_id = loaded_graph.get_tensor_by_name(\"movie_id:0\")\n",
    "    movie_categories = loaded_graph.get_tensor_by_name(\"movie_categories:0\")\n",
    "    movie_titles = loaded_graph.get_tensor_by_name(\"movie_titles:0\")\n",
    "    targets = loaded_graph.get_tensor_by_name(\"targets:0\")\n",
    "    dropout_keep_prob = loaded_graph.get_tensor_by_name(\"dropout_keep_prob:0\")\n",
    "    lr = loaded_graph.get_tensor_by_name(\"LearningRate:0\")\n",
    "    #两种不同计算预测评分的方案使用不同的name获取tensor inference\n",
    "#     inference = loaded_graph.get_tensor_by_name(\"inference/inference/BiasAdd:0\")\n",
    "    inference = loaded_graph.get_tensor_by_name(\"inference/ExpandDims:0\") # 之前是MatMul:0 因为inference代码修改了 这里也要修改 感谢网友 @清歌 指出问题\n",
    "    movie_combine_layer_flat = loaded_graph.get_tensor_by_name(\"movie_fc/Reshape:0\")\n",
    "    user_combine_layer_flat = loaded_graph.get_tensor_by_name(\"user_fc/Reshape:0\")\n",
    "    return uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, inference, movie_combine_layer_flat, user_combine_layer_flat\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 指定用户和电影进行评分\n",
    "这部分就是对网络做正向传播，计算得到预测的评分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rating_movie(user_id_val, movie_id_val):\n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "    \n",
    "        # Get Tensors from loaded model\n",
    "        uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, inference,_, __ = get_tensors(loaded_graph)  #loaded_graph\n",
    "    \n",
    "        categories = np.zeros([1, 18])\n",
    "        categories[0] = movies.values[movieid2idx[movie_id_val]][2]\n",
    "    \n",
    "        titles = np.zeros([1, sentences_size])\n",
    "        titles[0] = movies.values[movieid2idx[movie_id_val]][1]\n",
    "    \n",
    "        feed = {\n",
    "              uid: np.reshape(users.values[user_id_val-1][0], [1, 1]),\n",
    "              user_gender: np.reshape(users.values[user_id_val-1][1], [1, 1]),\n",
    "              user_age: np.reshape(users.values[user_id_val-1][2], [1, 1]),\n",
    "              user_job: np.reshape(users.values[user_id_val-1][3], [1, 1]),\n",
    "              movie_id: np.reshape(movies.values[movieid2idx[movie_id_val]][0], [1, 1]),\n",
    "              movie_categories: categories,  #x.take(6,1)\n",
    "              movie_titles: titles,  #x.take(5,1)\n",
    "              dropout_keep_prob: 1}\n",
    "    \n",
    "        # Get Prediction\n",
    "        inference_val = sess.run([inference], feed)  \n",
    "    \n",
    "        return (inference_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'load_dir' is not defined",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-29-13485381deef>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mrating_movie\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m234\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m1401\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m<ipython-input-28-ff91372f4c2a>\u001b[0m in \u001b[0;36mrating_movie\u001b[1;34m(user_id_val, movie_id_val)\u001b[0m\n\u001b[0;32m      3\u001b[0m     \u001b[1;32mwith\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mgraph\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mloaded_graph\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0msess\u001b[0m\u001b[1;33m:\u001b[0m  \u001b[1;31m#\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m         \u001b[1;31m# Load saved model\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m         \u001b[0mloader\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mimport_meta_graph\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mload_dir\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m'.meta'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m         \u001b[0mloader\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrestore\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msess\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mload_dir\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'load_dir' is not defined"
     ],
     "output_type": "error"
    }
   ],
   "source": [
    "rating_movie(234, 1401)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成Movie特征矩阵\n",
    "将训练好的电影特征组合成电影特征矩阵并保存到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n"
     ]
    }
   ],
   "source": [
    "loaded_graph = tf.Graph()  #\n",
    "movie_matrics = []\n",
    "with tf.Session(graph=loaded_graph) as sess:  #\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "    loader.restore(sess, load_dir)\n",
    "\n",
    "    # Get Tensors from loaded model\n",
    "    uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, _, movie_combine_layer_flat, __ = get_tensors(loaded_graph)  #loaded_graph\n",
    "\n",
    "    for item in movies.values:\n",
    "        categories = np.zeros([1, 18])\n",
    "        categories[0] = item.take(2)\n",
    "\n",
    "        titles = np.zeros([1, sentences_size])\n",
    "        titles[0] = item.take(1)\n",
    "\n",
    "        feed = {\n",
    "            movie_id: np.reshape(item.take(0), [1, 1]),\n",
    "            movie_categories: categories,  #x.take(6,1)\n",
    "            movie_titles: titles,  #x.take(5,1)\n",
    "            dropout_keep_prob: 1}\n",
    "\n",
    "        movie_combine_layer_flat_val = sess.run([movie_combine_layer_flat], feed)  \n",
    "        movie_matrics.append(movie_combine_layer_flat_val)\n",
    "\n",
    "pickle.dump((np.array(movie_matrics).reshape(-1, 200)), open('movie_matrics.p', 'wb'))\n",
    "movie_matrics = pickle.load(open('movie_matrics.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_matrics = pickle.load(open('movie_matrics.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成User特征矩阵\n",
    "将训练好的用户特征组合成用户特征矩阵并保存到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n"
     ]
    }
   ],
   "source": [
    "loaded_graph = tf.Graph()  #\n",
    "users_matrics = []\n",
    "with tf.Session(graph=loaded_graph) as sess:  #\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "    loader.restore(sess, load_dir)\n",
    "\n",
    "    # Get Tensors from loaded model\n",
    "    uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, _, __,user_combine_layer_flat = get_tensors(loaded_graph)  #loaded_graph\n",
    "\n",
    "    for item in users.values:\n",
    "\n",
    "        feed = {\n",
    "            uid: np.reshape(item.take(0), [1, 1]),\n",
    "            user_gender: np.reshape(item.take(1), [1, 1]),\n",
    "            user_age: np.reshape(item.take(2), [1, 1]),\n",
    "            user_job: np.reshape(item.take(3), [1, 1]),\n",
    "            dropout_keep_prob: 1}\n",
    "\n",
    "        user_combine_layer_flat_val = sess.run([user_combine_layer_flat], feed)  \n",
    "        users_matrics.append(user_combine_layer_flat_val)\n",
    "\n",
    "pickle.dump((np.array(users_matrics).reshape(-1, 200)), open('users_matrics.p', 'wb'))\n",
    "users_matrics = pickle.load(open('users_matrics.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_matrics = pickle.load(open('users_matrics.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 开始推荐电影\n",
    "使用生产的用户特征矩阵和电影特征矩阵做电影推荐"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 推荐同类型的电影\n",
    "思路是计算当前看的电影特征向量与整个电影特征矩阵的余弦相似度，取相似度最大的top_k个，这里加了些随机选择在里面，保证每次的推荐稍稍有些不同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recommend_same_type_movie(movie_id_val, top_k = 20):\n",
    "    \n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "        \n",
    "        norm_movie_matrics = tf.sqrt(tf.reduce_sum(tf.square(movie_matrics), 1, keep_dims=True))\n",
    "        normalized_movie_matrics = movie_matrics / norm_movie_matrics\n",
    "\n",
    "        #推荐同类型的电影\n",
    "        probs_embeddings = (movie_matrics[movieid2idx[movie_id_val]]).reshape([1, 200])\n",
    "        probs_similarity = tf.matmul(probs_embeddings, tf.transpose(normalized_movie_matrics))\n",
    "        sim = (probs_similarity.eval())\n",
    "    #     results = (-sim[0]).argsort()[0:top_k]\n",
    "    #     print(results)\n",
    "        \n",
    "        print(\"您看的电影是：{}\".format(movies_orig[movieid2idx[movie_id_val]]))\n",
    "        print(\"以下是给您的推荐：\")\n",
    "        p = np.squeeze(sim)\n",
    "        p[np.argsort(p)[:-top_k]] = 0\n",
    "        p = p / np.sum(p)\n",
    "        results = set()\n",
    "        while len(results) != 5:\n",
    "            c = np.random.choice(3883, 1, p=p)[0]\n",
    "            results.add(c)\n",
    "        for val in (results):\n",
    "            print(val)\n",
    "            print(movies_orig[val])\n",
    "        \n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n",
      "您看的电影是：[1401 'Ghosts of Mississippi (1996)' 'Drama']\n",
      "以下是给您的推荐：\n",
      "2371\n",
      "[2440 'Another Day in Paradise (1998)' 'Drama']\n",
      "2405\n",
      "[2474 'Color of Money, The (1986)' 'Drama']\n",
      "981\n",
      "[993 'Infinity (1996)' 'Drama']\n",
      "54\n",
      "[55 'Georgia (1995)' 'Drama']\n",
      "2205\n",
      "[2274 \"Lilian's Story (1995)\" 'Drama']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{54, 981, 2205, 2371, 2405}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommend_same_type_movie(1401, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 推荐您喜欢的电影\n",
    "思路是使用用户特征向量与电影特征矩阵计算所有电影的评分，取评分最高的top_k个，同样加了些随机选择部分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recommend_your_favorite_movie(user_id_val, top_k = 10):\n",
    "\n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "\n",
    "        #推荐您喜欢的电影\n",
    "        probs_embeddings = (users_matrics[user_id_val-1]).reshape([1, 200])\n",
    "\n",
    "        probs_similarity = tf.matmul(probs_embeddings, tf.transpose(movie_matrics))\n",
    "        sim = (probs_similarity.eval())\n",
    "    #     print(sim.shape)\n",
    "    #     results = (-sim[0]).argsort()[0:top_k]\n",
    "    #     print(results)\n",
    "        \n",
    "    #     sim_norm = probs_norm_similarity.eval()\n",
    "    #     print((-sim_norm[0]).argsort()[0:top_k])\n",
    "    \n",
    "        print(\"以下是给您的推荐：\")\n",
    "        p = np.squeeze(sim)\n",
    "        p[np.argsort(p)[:-top_k]] = 0\n",
    "        p = p / np.sum(p)\n",
    "        results = set()\n",
    "        while len(results) != 5:\n",
    "            c = np.random.choice(3883, 1, p=p)[0]\n",
    "            results.add(c)\n",
    "        for val in (results):\n",
    "            print(val)\n",
    "            print(movies_orig[val])\n",
    "\n",
    "        return results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n",
      "以下是给您的推荐：\n",
      "930\n",
      "[942 'Laura (1944)' 'Crime|Film-Noir|Mystery']\n",
      "910\n",
      "[922 'Sunset Blvd. (a.k.a. Sunset Boulevard) (1950)' 'Film-Noir']\n",
      "847\n",
      "[858 'Godfather, The (1972)' 'Action|Crime|Drama']\n",
      "598\n",
      "[602 'Great Day in Harlem, A (1994)' 'Documentary']\n",
      "1179\n",
      "[1197 'Princess Bride, The (1987)' 'Action|Adventure|Comedy|Romance']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{598, 847, 910, 930, 1179}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommend_your_favorite_movie(234, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 看过这个电影的人还看了（喜欢）哪些电影\n",
    "- 首先选出喜欢某个电影的top_k个人，得到这几个人的用户特征向量。\n",
    "- 然后计算这几个人对所有电影的评分\n",
    "- 选择每个人评分最高的电影作为推荐\n",
    "- 同样加入了随机选择"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def recommend_other_favorite_movie(movie_id_val, top_k = 20):\n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "\n",
    "        probs_movie_embeddings = (movie_matrics[movieid2idx[movie_id_val]]).reshape([1, 200])\n",
    "        probs_user_favorite_similarity = tf.matmul(probs_movie_embeddings, tf.transpose(users_matrics))\n",
    "        favorite_user_id = np.argsort(probs_user_favorite_similarity.eval())[0][-top_k:]\n",
    "    #     print(normalized_users_matrics.eval().shape)\n",
    "    #     print(probs_user_favorite_similarity.eval()[0][favorite_user_id])\n",
    "    #     print(favorite_user_id.shape)\n",
    "    \n",
    "        print(\"您看的电影是：{}\".format(movies_orig[movieid2idx[movie_id_val]]))\n",
    "        \n",
    "        print(\"喜欢看这个电影的人是：{}\".format(users_orig[favorite_user_id-1]))\n",
    "        probs_users_embeddings = (users_matrics[favorite_user_id-1]).reshape([-1, 200])\n",
    "        probs_similarity = tf.matmul(probs_users_embeddings, tf.transpose(movie_matrics))\n",
    "        sim = (probs_similarity.eval())\n",
    "    #     results = (-sim[0]).argsort()[0:top_k]\n",
    "    #     print(results)\n",
    "    \n",
    "    #     print(sim.shape)\n",
    "    #     print(np.argmax(sim, 1))\n",
    "        p = np.argmax(sim, 1)\n",
    "        print(\"喜欢看这个电影的人还喜欢看：\")\n",
    "\n",
    "        results = set()\n",
    "        while len(results) != 5:\n",
    "            c = p[random.randrange(top_k)]\n",
    "            results.add(c)\n",
    "        for val in (results):\n",
    "            print(val)\n",
    "            print(movies_orig[val])\n",
    "        \n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n",
      "您看的电影是：[1401 'Ghosts of Mississippi (1996)' 'Drama']\n",
      "喜欢看这个电影的人是：[[5438 'M' 25 14]\n",
      " [3901 'M' 18 14]\n",
      " [5102 'M' 25 12]\n",
      " [1131 'M' 56 13]\n",
      " [5055 'F' 35 16]\n",
      " [4506 'M' 50 16]\n",
      " [1763 'M' 35 7]\n",
      " [4903 'M' 35 12]\n",
      " [3031 'M' 18 4]\n",
      " [1130 'M' 18 7]\n",
      " [3557 'M' 18 5]\n",
      " [85 'M' 18 4]\n",
      " [982 'F' 25 9]\n",
      " [3008 'M' 18 4]\n",
      " [3833 'M' 25 1]\n",
      " [660 'M' 45 16]\n",
      " [100 'M' 35 17]\n",
      " [4085 'F' 25 6]\n",
      " [5861 'F' 50 1]\n",
      " [371 'M' 18 4]]\n",
      "喜欢看这个电影的人还喜欢看：\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-42-020dd0a9364a>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mrecommend_other_favorite_movie\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m1401\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m20\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m<ipython-input-41-0e800bd93a54>\u001b[0m in \u001b[0;36mrecommend_other_favorite_movie\u001b[1;34m(movie_id_val, top_k)\u001b[0m\n\u001b[0;32m     31\u001b[0m         \u001b[0mresults\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     32\u001b[0m         \u001b[1;32mwhile\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mresults\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m!=\u001b[0m \u001b[1;36m5\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 33\u001b[1;33m             \u001b[0mc\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mp\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mrandom\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrandrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtop_k\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     34\u001b[0m             \u001b[0mresults\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0madd\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     35\u001b[0m         \u001b[1;32mfor\u001b[0m \u001b[0mval\u001b[0m \u001b[1;32min\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mresults\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ],
     "output_type": "error"
    }
   ],
   "source": [
    "recommend_other_favorite_movie(1401, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 结论\n",
    "\n",
    "以上就是实现的常用的推荐功能，将网络模型作为回归问题进行训练，得到训练好的用户特征矩阵和电影特征矩阵进行推荐。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 扩展阅读\n",
    "如果你对个性化推荐感兴趣，以下资料建议你看看："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [`Understanding Convolutional Neural Networks for NLP`](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/)\n",
    "\n",
    "- [`Convolutional Neural Networks for Sentence Classification`](https://github.com/yoonkim/CNN_sentence)\n",
    "\n",
    "- [`利用TensorFlow实现卷积神经网络做文本分类`](http://www.jianshu.com/p/ed3eac3dcb39?from=singlemessage)\n",
    "\n",
    "- [`Convolutional Neural Network for Text Classification in Tensorflow`](https://github.com/dennybritz/cnn-text-classification-tf)\n",
    "\n",
    "- [`SVD Implement Recommendation systems`](https://github.com/songgc/TF-recomm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "今天的分享就到这里，请多指教！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PY36-tf12",
   "language": "python",
   "name": "py36_b"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
